EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

What it claims

Emotion steering vectors (mean activation of emotional speech minus mean of neutral speech) contain two separable parts. One is a shared shift from neutral toward the centroid of all emotions, which makes speech sound "emotional". The other is a per-emotion residual from that centroid, which determines which emotion is expressed. Scaling the residual independently of the shared part, without any training, gives better emotion adherence than CoCoEmo's single global strength.

On held-out IEMOCAP, listeners' majority label matched the dominant target emotion:

• IndexTTS-2: 73.89% of clips with EmoRES vs 54.72% with CoCoEmo.
• CosyVoice2: 77.88% vs 58.18%.

How it works

1. Extract vectors. This is offline and gradient-free. For each emotion e, v_e = h̄_e − h̄_neu is computed from paired neutral/emotional utterances with the same speaker and text, drawn from ESD, CREMA-D and RAVDESS. A pair is dropped whole if either side fails an SER gate or a signal-quality gate.
2. Decompose. With centroid h̄_c (the unweighted mean over emotions), v_e = (h̄_c − h̄_neu) + (h̄_e − h̄_c).
3. Reweight. v_RES = λ_c·shared + λ_r·residual. Setting λ_c = λ_r = 1 recovers CoCoEmo exactly, so it is a strict generalisation.
4. Mixed emotions. Replace h̄_e with h̄_p = Σ p_e·h̄_e. The shared term stays fixed, and only the residual varies with the requested proportions.
5. Inject. Compute h ← h + αv at the attention output, then rescale to the original ‖h‖ so that only direction changes.

flowchart LR
  A[Emotion means] --> C[Centroid]
  N[Neutral mean] --> S[Shared: centroid − neutral]
  C --> S
  C --> R[Residual: emotion − centroid]
  A --> R
  S -->|λc| V[Steering vector]
  R -->|λr, the lever| V
  V --> H[Add at attn output, restore norm]

Evidence

Backbones and layers steered. Both backbones are frozen, with native emotion conditioning left at its default.

• IndexTTS-2 (embedding-conditioned): layers 1, 6 and 8 of its semantic LM.
• CosyVoice2 (instruction-conditioned): layers 14 and 17 of its text-to-token LM.

These layers are the top-K separable layers published by CoCoEmo, not re-searched.

Fair comparison. All methods' vectors are rescaled to the same norm. At a fixed α, the conditions differ only in direction.

Tuning and test split. All tuning was done on CREMA-D. IEMOCAP is held out entirely.

Objective results, IndexTTS-2 / CosyVoice2 vs CoCoEmo:

The paper says EmoRES wins on all four objective emotion metrics (TEP, E-SIM, ρ, H-Rate).

Human evaluation (at least 9 annotators per sample):

• Dom-hit rises as quoted above.
• Fidelity improves by up to 17.3% relative.
• Pairwise naturalness preference for EmoRES is 63.80% on IndexTTS-2 and 60.32% on CosyVoice2, reported as statistically significant.

Ablation. The best control came from keeping the shared component while boosting the residual.

How to use it

Code is at https://github.com/facebookresearch/EmoRES-TTS. The fetched text gives no install commands, licence, or chosen λ_c/λ_r values. To adopt it you need:

• the paired extraction corpora,
• a supported backbone,
• the CoCoEmo layer choices above.

Limits

• Only four emotions are covered (angry, happy, sad, surprise), with neutral as the reference.
• The CREMA-D development set annotates only angry, happy and sad. Surprise was therefore never tuned on, only tested.
• The only baseline is CoCoEmo. There is no comparison against native conditioning alone or other steering methods such as EmoSteer-TTS.
• Speaker similarity and WER are listed as metrics, but the fetched text does not report their values. Check the full paper for intelligibility and speaker-identity cost before relying on it.
• The page text is truncated partway through the metrics section. Appendix details (extraction settings, coefficient values, geometric interpretation) were not available.