CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

What it claims

CARES fixes each sound's salience label before any audio exists, then generates the scenes to match. It contains 10,000 two-speaker English scenes, 120.4 ± 18.8 s long on average. They are built from 1,000 templates covering 24 adult role pairs, 13 locations and 219 distinct sounds. Each sound carries one of four reaction labels:

• pivot: the sound is named and the conversation reorganizes around it.
• verbal: the sound is named in passing, then the talk resumes.
• behavioral: the sound is never named, but a speaker breaks off, restarts or is interrupted.
• ambient: no reaction at all.

The headline result is that six audio-language models classify reactions barely above chance (25%). The best, Qwen3-Omni, reaches 39.0% balanced accuracy. A text-only Whisper + LLM control reaches 36.5%.

How it works

The paper uses a deterministic allocator and a fixed set of models:

• Allocator. It samples scene, relation, speaker genders, events, reaction register and composition uniformly. A scene has 0–3 events, with at most one pivot. Event order is shuffled so that position does not encode type. The authors pre-allocate because a language model choosing these itself would collapse onto a few familiar modes.
• Topics come from Qwen3-235B-Instruct.
• Dialogues (up to 13 utterances) come from Claude Opus 5.
• Evaluator is Llama-4-Maverick, a different model family from the dialogue writer.
• Speech is synthesized with ElevenLabs v3.

flowchart LR
  A[Deterministic allocator] --> B[LLM writes topic + dialogue]
  B --> C{Evaluator recovers topic and every reaction?}
  C -- no --> B
  C -- yes --> D[TTS + forced alignment]
  E[Freesound events + ambience] --> F[Mix, RIR, loudness]
  D --> F

Acceptance filter. The evaluator reads the transcript with each sound marked as [SOUND: <id>]. A dialogue is accepted only if:

• every sound's four-way reaction label is recovered exactly, and
• the recovered topic matches the gold topic, with BGE-large cosine similarity ≥ 0.82.

The 0.82 threshold sits at the max-Youden point (ROC AUC 0.861). Rejected dialogues are regenerated.

Mixing. The mix is designed so that timing and level do not leak the label:

• Every word or turn boundary gets a gap drawn from the same range, whether or not an event falls in it.
• Each event sits at least 6 dB above the ambience.
• Loudness follows EBU R128: events at −20 or −23 LUFS (chosen at random), ambience at −32 LUFS, master at −23 LUFS.
• Behavioral sounds end just before speech resumes and duck the reacting voice by up to 4 dB.

Evidence

Label validity. Three Fiverr annotators labelled 200 test scenes from transcripts. They agreed with the released labels at κ = 0.86 and with each other at κ = 0.81.

Intelligibility. Whisper-large-v3 gets 3.3% WER on the final mix (95% CI 2.8–3.9). For comparison, it gets 2.7% on LibriSpeech test-clean.

Benchmark. All results are zero-shot on the 1,000 test scenes, with chance at 25%:

Scene classification and tagging are four-way multiple choice. The models recognize scenes and sounds well, but reaction classification stays near chance.

Per-class collapse. Several models effectively predict a single class:

• MOSS-Audio has 99.4% pivot recall.
• Kimi-Audio has 94.1% ambient recall.
• MiMo-Audio and MiDashengLM never predict behavioral or ambient.

Summary recall. In 200-word summaries, behavioral sounds are mentioned at most 1.6% of the time. The best category is pivot sounds for Qwen3-Omni, at 22.2%.

How to use it

• The code and the full generation prompts are in the GitHub repo marcelgibier/CARES.
• Audio is 44.1 kHz mono 16-bit WAV.
• Splits are 80/10/10 by template, so no situation is shared across splits.
• Test-set event takes, ambiences and room impulse responses are held out from train and validation.
• Train and validation splits are released for fine-tuning and probing. The benchmark itself is zero-shot.

Limits

Scope of the data:

• Speech is synthetic TTS in English only, between two adults.
• Each scene has at most 3 events.
• Salience is defined only for audio events, and only through audible speaker reaction.

Validation:

• Labels were checked on transcripts, not on the audio.
• The 2.5-point gap between Qwen3-Omni and the text-only control is close to the ±2.0 confidence interval.

Missing information:

• The fetched text cuts off mid-way through the benchmark task definitions.
• No dataset licence is stated, including for the Freesound assets. Check the repository before redistributing.