MUB: An Identity-Masked Benchmark for Music Understanding in Multimodal Language Models

What it claims

MUB packages 500 recordings with four target outputs each: scalar tempo, tonic and mode, time signature, and full-track structural intervals. The authors (Kun Liu and Lyra Tao, October 2026) frame it as a resource and protocol under validation. They concede that GaMMA, MusTBench, MuChoMusic and HumMusQA already cover these task areas. The claimed contribution is the combination of identity masking, scalar global labels and full-recording interval outputs.

How it works

Labels come from an existing professional annotation archive with 2,345 task records. The builder takes the last annotation submission for each record and derives global labels as the plurality across segments. Among the 500 selected entries, 399 have one recorded operator and 117 have one recorded submission. No quality-review status is enforced.

Eligible candidates have all three global labels, at least five intervals and a duration of 90 to 600 seconds. Deduplicating by title and artist leaves 1,111 candidates. Sampling stratifies on four tempo bins crossed with key and mode, using seed 42, and is not proportional. Metre, genre and duration are not strata.

The masking protocol separates evaluator-side data from model input:

flowchart LR
  R[Prompt JSONL record] --> A[Inference adapter]
  A -->|forward| M[Model: audio + task text]
  A -.->|must strip| H[audio_hash, gold labels]
  A -.->|routing only| I[MUB-00001..00500 ID]

The audio_hash field is SHA-256 of the first 1,048,576 bytes of the compressed file, truncated to 16 hex characters. The prompt records currently include it, so the adapter must drop it. Embedded audio tags and filenames are not certified as sanitized.

Evidence

Results are for two reference systems only. No multimodal LLM has been run.

The DSP baseline decodes audio to 22,050 Hz mono. It estimates tempo by spectral-flux autocorrelation over 60 to 180 BPM and key with Krumhansl-style profiles. Its metre heuristic cannot output 6/8. Neither system predicts intervals, so the zero F1 reflects abstention.

Metadata statistics: median tempo is 101 BPM (interquartile range 75 to 128). Median duration is 228 s. Recordings have a median of 9 intervals. There are 34 distinct lowercased section labels. Metre is 492 recordings in 4/4, 4 in 3/4 and 4 in 6/8.

How to use it

Code and the manifest are at https://github.com/Token-Spark/music-understanding-benchmark. Audio is not distributed. Matching uses a byte prefix, so a re-encoded copy of a recording will not match, and the scripts do not download missing audio. External reproduction needs a private ID mapping and a legitimate way to access the audio.

No harness enforces the no-tools rule. The deployment itself must disable search, downloads, code execution and model-callable tools. The tempo prompt asks for an integer from 40 to 220. The authors require logging of the model version, decoding and resampling, chunking, retries, refusal rates and tool configuration.

Limits

• Key metric bug: the key scorer has a directional bug. It accepts A minor against C major but rejects the reverse. Re-scoring with a symmetric relative-key rule gives 29.4% (DSP) and 11.0% (constant). Those figures are diagnostic recalculations, not output of a fixed scorer.
• Submission scoring bug: the evaluator excludes missing IDs from the denominator and counts duplicate IDs more than once. This makes it unsafe for open submissions. The two stored baselines are unaffected.
• Segmentation scoring: it uses greedy matching at IoU ≥ 0.5, ignores section labels and does not validate intervals.
• Tempo scoring: the tempo metric gives no credit for half or double tempo. Single labels flatten modulation and metre changes.
• Contamination: masking removes titles but cannot rule out training-set exposure. The partial hash could identify a recording against a known catalogue.
• Rights: deduplication does not guarantee unique audio. No licence field was used, and rights clearance for the recordings is unresolved.