SAID: Semantic Acoustic Imaging Detector for Sound Event Localization and Detection

What it claims

SAID turns four-channel 48 kHz audio into up to 16 labeled 180×360 acoustic maps per frame at 10 fps. Each map carries a source's region, its energy pattern, a class from 13 DCASE classes, and a confidence. The two pieces are an encoder, Audio2Sph, which arranges audio features on a direction grid, and a decoder, Sph2Imaging, which is a Mask2Former-style slot decoder. The submitted system ranked first on the DCASE2026 Task 3 Track A evaluation set with 0.1080 Macro mAP and 0.3962 Macro Pearson r. Code and demos are at https://github.com/IN03X/SAID.

How it works

Encoder (Audio2Sph).

• STFT front end. A 2,048-sample Hann window with a 480-sample hop gives 100 fps. The input is multichannel log-magnitude plus phase sin/cos, fed to four ConvNeXt stages. The last three stages produce features at three scales.
• Spherical Cross-Attention. This is the step that maps frequency to direction. A learned query sits at each cell of a 45×90 (elevation × azimuth) grid, with D=16. Each query attends over frequency positions. The three scales have separate blocks that share queries, and their outputs are summed.
• Time Alignment. Average- and max-pooling reduce the frame rate to 10 fps.

Pretraining head. During pretraining only, a Panoramic Decoder predicts one class-agnostic 180×360 map per frame at 100 fps, and Time Alignment is skipped. The decoder uses convolutions, bilinear upsampling and a sigmoid. It is discarded afterwards.

Decoder (Sph2Imaging).

• Multi-scale features. Multi-Scale Deformable Attention and Up Blocks build Small, Medium and Large grids with S=256 channels.
• Class features. A frozen PaSST-S, initialised from AudioSet and then trained further on DCASE, supplies class features. These are combined with 13 learned class embeddings.
• Slot decoding. Sixteen slot queries pass through a nine-layer mask decoder that cycles Large → Medium → Small grids. Each layer applies masked cross-attention, then cross-attention to the PaSST features, then self-attention across slots.
• Masking rule. The attention mask is the previous layer's map thresholded at 0.5. If that mask is empty, the layer attends to the full grid.
• Heads.
• The Active Head outputs 13 classes plus no-object. A slot's activity is 1 − p(no-object).
• A separate Class Head cross-attends to ConvNeXt features and gives the output class.
• A Refine Block upsamples maps from 45×90 to 180×360.
• Confidence. A slot's confidence is the product of three terms: activity, the top class probability, and the mean refined-map value over pixels above 0.5.

flowchart LR
  A[4-ch audio] --> E[Audio2Sph]
  E -->|pretraining only| P[Panoramic Decoder: class-agnostic map]
  E --> T[Time Alignment 10 fps] --> D[Sph2Imaging slot decoder]
  C[Frozen PaSST class features] --> D
  D --> O[16 slots: map + active + class]

Training, in three stages:

1. Pretrain Audio2Sph for 2.5M steps at batch size 1. Scenes are simulated online with Pyroomacoustics ISM and contain 0–4 VCTK sources. The target at each frame is the maximum of spherical Gaussians (σ = 4°) centred on active sources. The loss is BCE.
2. Train full SAID on SourceBank. SourceBank is 122,359 clips (about 95.4 h) from VCTK, MUSDB18-HQ, FSD50K and FSDKaggle2018, mapped to the 13 classes, with 1–6 sources per scene. - Extended sources. These are simulated as several emitting points driven by the same clip. - Matching. Hungarian matching is done per frame and per decoder group. - Loss. The objective is 2·CE + 5·BCE + 5·Dice + L_refine + focal. L_refine includes Pearson-correlation and soft-IoU terms. The focal loss uses γ=2 and inverse-sqrt class-frequency weights clipped to [0.25, 4].
3. Fine-tune on real DCASE recordings with the same loss. Augmentation applies azimuth rotations of 0/90/180/270°, made by permuting microphone configurations and shifting the target maps.

Evidence

• Setting. The DCASE2026 Task 3 Track A evaluation set has 79 recordings (about 3.5 h) and 13 classes. Systems use audio only, with no video.
• Metrics.
• Macro mAP scores class plus region overlap across thresholds and penalises misses and false detections.
• Macro Pearson r scores energy-pattern correlation for matched predictions of the same class.
• The official ranking is the sum of the two metric ranks.
• Result. First place with 0.1080 / 0.3962.

The fetched text gives no numbers for the Mask R-CNN baseline or for other entries, and no ablations, so the size of the margin cannot be judged from this page.

Submission compression. Refined maps are stored as sparse points to meet the 20 MB-per-recording limit.

• Each grid cell keeps its strongest pixel at or above 10% of the map peak.
• Grid spacing is 2 px when confidence is at least 0.20 and 6 px otherwise.
• Lower-confidence maps also get boundary support points at 12% of peak.
• All detections are kept.

Limits

• Truncated text. The page cuts off mid-Section 4, so any ablations, the baseline comparison and the stated limitations are missing.
• Absolute scores are low. A Macro mAP of 0.108 is a winning score, but in absolute terms the task remains far from solved.
• Simulation gap. Pretraining and SourceBank data come from ISM simulation of point sources. Extended sources are only approximated by groups of points.
• Fixed input format. Inputs are 2-second, four-channel segments in the DCASE microphone format. The rotation augmentation depends on that array geometry.
• Pretraining cost. 2.5M pretraining steps is a substantial budget. No hardware or wall-clock figures are given.
• Class dependency. Classification relies on a PaSST model fine-tuned on DCASE data and then frozen, so it is tied to the 13 DCASE classes.
• Licence. The page states no licence for the code or weights.