Benchmarking color judgment in multimodal LLMs with CIE-computable ground truth

What it claims

The benchmark has five color tasks and computes every answer from CIE formulas, with no human labels. Eight MLLM endpoints were tested in April 2026: four Qwen3-VL variants, GLM-4.5V, GPT-5.4, Claude-Opus-4.7 and Gemini-3.1-Pro. The authors (Yue, Xiao, Wang, Chen) find that models mostly pass coarse binary white-balance checks. They are poorly calibrated on numeric CCT and on fine ΔE2000 judgments. The authors attribute the failures to five distinct patterns, each tied to a model family, the open-weight cluster or a single model. The closed-model CCT advantage shrinks when the ColorChecker template is removed.

How it works

Each task uses a fixed zero-shot prompt that returns single-line JSON. Calls use temperature 0, no in-context examples, and 480×320 PNG stimuli.

• A, AWB pair (24 pairs): two ColorChecker-24 charts, one D65-adapted and one with an R/B/G/M cast gain in linear sRGB. The model picks the correct image and names the cast.
• B, skin-tone bias (48): a dark-skin or light-skin patch is shifted in CIELab along one of 8 hue directions at ΔE of about 4, 8 or 14. The model names the direction and size.
• C, CCT (27): the chart is rendered under 9 white points from 2700 to 10000 K, with 3 noise seeds each. Below 3500 K the white point comes from the Planckian locus, and from 4200 K up it comes from the CIE D locus.
• D, ΔE2000 ordering (24): the model picks which of two patch pairs differs more. Gaps are at least 4 ΔE units.
• E, naming under illuminant (40): 8 patches are rendered without adaptation at 5 CCTs and named with the 11 Berlin & Kay terms. The reference name is the nearest ISCC-NBS centroid in CIELab.

Rendering stores sRGB patch values under D65 rather than spectra. A Bradford CAT maps them to the scene white. The pipeline then adds σ=0.01 gain noise and clips at zero only.

Evidence

Intervals are 1000-resample bootstrap 95% CIs.

• CCT MAE: GPT-5.4 scores 715 K, Claude 978 K and Gemini 1056 K. GLM-4.5V scores 1585 K, and Qwen ranges from 1358 K (32B-Thinking) to 2133 K (30B-A3B MoE). Claude has the best within-500 K rate (0.56) against 0.11 for Qwen-8B.
• Daylight anchor: the four Qwen variants and GLM cluster their CCT answers in a daylight band. Qwen-32B-Instruct answers 5500 K for all 27 stimuli.
• Non-ColorChecker control (Task C, 24 random sRGB patches): GPT-5.4 degrades from 715 K to 1417 K. Gemini goes from 1056 K to 1793 K, and Qwen-32B-Thinking from 1358 K to 2189 K. Claude improves from 978 K to 752 K, and Qwen-32B-Instruct stays at 1922 K.
• ΔE ordering (chance 0.50): all four Qwen models plateau at 0.75 to 0.79. GLM, Claude and Gemini reach 0.875, and GPT-5.4 reaches 0.917. On the 4 triplets where linear-sRGB distance and ΔE2000 disagree, no model follows RGB consistently.
• Warm-bias collapse: Qwen-8B and Qwen-30B-MoE answer "warm" every time on A-bias and score 0.25 (chance). The other models reach 0.92 to 1.00.
• Skin-tone direction (chance 0.375): only GPT-5.4 (0.79) and Gemini (0.75) are clearly above chance. Cool shifts are named red or yellow.
• Naming: raw constancy ranges from 0.68 to 0.85. It is conditioned on a correct name at 6500 K, so 7 of 8 models tie at 0.733 and Qwen-30B-MoE scores 0.500. Most models misname yellow, purple and brown even at D65.
• Claude-Opus-4.7: it names the cast correctly on every trial (1.00) but picks the correct image only 0.667 of the time. Errors split 4/4 across positions, which rules out a position bias.

How to use it

The benchmark, raw responses with call timestamps, and analysis scripts are released, but the page gives no URL or licence. It runs against any OpenAI-compatible endpoint with no model-specific code. The authors used max tokens 512 to fit the Thinking variant's reasoning. They reached Qwen and GLM through SiliconFlow and closed models through OpenRouter, using openai/gpt-5.4, anthropic/claude-opus-4.7 and google/gemini-3.1-pro-preview. They suggest using GPT-5.4 and Gemini-3.1-Pro, both perfect on Task A, only as coarse screening checks where no physical ground truth exists. They say current MLLMs cannot replace camera-characterisation tools for fine color work.

Limits

• Small samples: each task has 24 to 48 stimuli, and Task E conditional constancy rests on only 2 or 3 qualifying patches per model.
• No spectra: the patches are sRGB values, not reflectance spectra. Version 2 is planned to use measured BabelColor spectra.
• Confounded causes: the warm-bias collapse cannot be separated into a Qwen-family effect versus a low-active-parameter effect. The source of the daylight anchor is untested. The Claude asymmetry is reported only as an observation.
• Moving targets: closed endpoints can change without notice, and each task uses a single prompt template.
• Truncated page: the fetched text ends partway through the Discussion section.