Improving the Dependency on Fine-Tuning Data in Transfer Learning for Speech Emotion Recognition

What it claims

A feature maximum likelihood linear regression (fMLLR) block placed inside Wav2vec 2.0, between the CNN feature extractor and the transformer layers, compensates for speaker and room mismatch in speech emotion recognition (SER). It predicts arousal, valence and dominance. On a robot-recorded test set, it raises mean concordance correlation coefficient (CCC) by 16.8% over plain Wav2vec 2.0 when fine-tuned on 12.9% of the training data, and by 14.4% with 50%. The authors say the same block applies directly to HuBERT and WavLM, though only Wav2vec 2.0 is tested.

How it works

fMLLR estimates an affine transform x' = Ax + b by expectation-maximization. Each row needs the inverse of an (L+1)×(L+1) matrix, which becomes singular at the 512 dimensions of the Wav2vec CNN output. The design follows from that constraint.

1. A convolutional autoencoder compresses the 512-dim CNN features through 256, 128 and 64 to a 40-dim latent space. It uses kernel 3, padding 1, batch norm in every layer and ReLU, with a symmetric decoder. The authors tried latent sizes from 16 to 64 and picked 40 as the balance between compression and fMLLR stability.
2. The autoencoder is trained with an MSE reconstruction loss on simulated data passed through frozen Wav2vec CNN layers. Training runs for 100 epochs at batch size 64 with Adam, ReduceLROnPlateau and early stopping after 5 epochs.
3. fMLLR uses a single GMM rather than an HMM-GMM, so it needs no language model or transcripts and runs unsupervised. The GMM uses diagonal covariances, up to 200 EM iterations and variance regularization of 1e-3.
4. GMM size was screened by BIC over 2 to 128 mixtures. BIC kept improving up to 128, but matrix inversion grew unstable as mixtures increased, so the final count was chosen on validation performance and stability. The fetched text does not state the chosen number.

Both domains are pushed toward one space, as below.

flowchart LR
  A[Clean MSP-Podcast train] -->|RIRs + noise + MVDR| S[Simulated domain]
  B[Real HRI recordings] -->|MVDR, then fMLLR in latent space| S
  S --> T[Wav2vec transformer + MLP]

Training audio is convolved with measured room impulse responses (RIRs), mixed with noise and beamformed with MVDR. At inference, real robot audio is mapped toward that simulated space by fMLLR.

Evidence

Training data is MSP-Podcast v1.9: 55,283 training, 9,546 validation and 21,560 test turns, 137 hours in total. The test set is HRI-static, from earlier work by Grágeda et al. It is the MSP-Podcast test partition replayed to a PR2 robot with a head-mounted Microsoft Kinect, over 32 hours of audio. The speaker sits 2 m away at 0°, with restaurant noise sources at ±45° and SNR of 5 dB. Simulation used three sets of 63 RIRs per Kinect microphone, measured at 1, 2 and 3 m over 21 head angles.

The authors cite a 22.5% performance gap that remained between earlier simulation-trained HRI systems and matched conditions. This work targets that gap.

Limits

The fetched text breaks off at the start of the systems section, so the results tables, per-attribute CCC, absolute values and ablations are not visible. Only the abstract's relative gains are available. Evaluation covers one static setup: one distance, one noise type and one SNR. Training assumes the direction of arrival is known, estimated by computer vision. Results depend on stable matrix inversion, which limits both latent size and GMM size. No code, weights or licence are mentioned.