RMS-AQA: A Two-Stage Spatial Audio Question Answering Benchmark for Real-World Domestic Environments

What it claims

RMS-AQA is a spatial audio QA benchmark, selected as an ICASSP 2027 Grand Challenge (https://rmsaqachallenge.github.io/). It has 177K ten-second, 24 kHz first-order Ambisonics (FOA) clips and 179K two-stage, four-option QA pairs. The headline metric is grounded Stage-2 accuracy (A_GS2): a sample counts only if both the Stage-1 event-grounding answer and the Stage-2 reasoning answer are correct. Raw Stage-2 accuracy can therefore be credited even when grounding is wrong.

The authors' FOA plug-in on a frozen Qwen3-Omni reaches 48.93% A_GS2 on validation. That beats Qwen3.8-Omni-Flash, a closed model with native FOA input, at 36.38%.

How it works

Data splits (rooms are disjoint across splits):

• Recorded audio: a Zoom H3-VR in 18 real rooms, 49 sound classes, up to four simultaneous sources. Unsafe sounds were replayed through loudspeakers. Annotations were done manually.
• Simulated audio: made with SpatialScaper. Dry sources come from AudioSet, FSD50K, ESC-50 and CHiME-Home, filtered to 324 domestic classes. They are convolved with measured FOA RIRs from FOA-MEIR, METU-SPARG, TAU-SRIR and ARNI-SRIR, covering 1,396 rooms.
• Distribution shift between splits:
• Moving sources: 45.04% of train, 75.70% of simulated validation, 34.36% of recorded validation.
• Maximum source distance: 7.2 m in train, 6.4 m in simulated validation, 5.9 m in recorded validation.
• QA generation: GPT-5.5 writes questions from metadata, Gemini 3.1 paraphrases them, and humans verify them. Human verification is stated for the validation set only.
• Question categories: six, balanced. They are sound counting, spatial location, temporal detection, spatial relation, temporal relation, and action prediction (what response the situation warrants).

Baseline architecture. Two branches converge on a zero-initialised projection:

flowchart LR
  W[FOA W channel log-Mel] --> AE[Frozen audio encoder]
  F[4ch log-Mel + 3 intensity vectors] --> SE[Spatial encoder: ResNet-18 + 8 Conformers]
  AE --> C[Concat + zero-init linear]
  SE --> C
  C --> L[Frozen LLM + LoRA]
  L --> S1[Stage 1 answer]
  S1 --> L
  L --> S2[Stage 2 answer]

• The spatial encoder uses a ResNet-18 that pools only along frequency, so the time axis is preserved. A linear projection maps to 1024 dimensions, followed by 8 Conformer blocks and a LayerNorm.
• The projection is zero-initialised, so training starts from the semantic-only mapping.
• Only three things are trained: the spatial encoder, the fusion projection, and the LoRA adapters.

Training and inference:

• LoRA: rank 16, α = 16, dropout 0.05, on all 28 LLM layers.
• Optimiser: AdamW, learning rate 2e-4, weight decay 1e-2, gradient clipping 5.0.
• Schedule and hardware: 3 epochs on 4× A100, batch size 8 per GPU.
• Stage-2 conditioning: during training, Stage 2 is conditioned on the reference Stage-1 answer. At inference it uses the predicted Stage-1 answer, decoded greedily with up to 48 tokens.

Evidence

All numbers are A_GS2 on the validation set, comparing zero-shot to "+ SpatialAug":

Gains range from 23.32 to 35.28 points. Closed zero-shot baselines:

• Qwen3.8-Omni-Flash: 36.38
• Gemini-3.1-Pro: 29.55
• GPT-Audio-1.5: 17.07

Two patterns matter for anyone using the benchmark:

• Raw Stage-2 accuracy overstates reasoning. For adapted Qwen3-Omni, Stage-1 accuracy is 66.70 and raw Stage-2 is 63.45, but A_GS2 is 48.93. Zero-shot MiDashengLM scores 34.97 raw Stage-2 but only 9.40 A_GS2.
• Spatial location is the weakest category even after adaptation. The best grounded scores are spatial location 31.40 and spatial relation 38.30, against 60.90 for action prediction (all adapted Qwen3-Omni).

How to use it

The challenge site is the stated entry point. The fetched text gives no download instructions, code, weights release or licence. To reproduce the baseline, the page supplies the hyperparameters listed above.

Limits

• Results are validation-only, and validation also selected the checkpoint. The checkpoint with the best validation A_GS2 was used for evaluation. The fetched text ends at the Table 3 ablation, and no recorded-test numbers appear.
• The test set is small: 5 recorded rooms and 3K QA pairs.
• Training QA is LLM-generated and not stated to be human-verified.
• Training is entirely simulated while test is entirely recorded. The authors name this sim-to-real gap as a main failure mode, alongside concurrent sources and far distance.
• No licence or usage terms are given. Recorded media sources are described only as "public-domain".