Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning

What it claims

DecoyCount is a 1,660-item counting benchmark from Elorian and Mercor. It tests whether vision-language models count what is visible or substitute expectations such as reflections, tray slots or pack labels. The accompanying paper concludes that "stronger general reasoning does not by itself generalize to aligned visual reasoning." It evaluates six frontier models, all below 50%. Elorian's own model, Glimpse, launched October 2026, is not in the paper. The 50.48% figure for Glimpse comes only from CEO Andrew Dai, by email.

How it works

Dai says the benchmark separates two error types that final-number scoring merges. His example is an apple in front of a mirror. Missing the apple is a vision failure. Counting the apple and its reflection as two is a reasoning failure. Scoring both the same way hides which one to fix.

Glimpse is built on "visual thinking": the model generates and edits images as steps in its reasoning trace, alongside language. The generated image is part of the chain of reasoning, not only an output. One example Dai gives is crossing off people one by one while counting around a table. Dai contrasts this with world models such as World Labs' Atlas and Google's Genie 4, which build simulated 3D environments for "direct action." Glimpse targets "indirect action": informing a decision, such as whether a video violates a policy.

Evidence

From the paper, each model's score is averaged over three runs under its reported settings. GPT-6 Astra scores 43.17%, Gemini-3.8-flash 32.71%, Qwen3.8-max 28.31% and Claude-Fable-5.1 25.22%. The page names only four of the six models.

Glimpse's 50.48% is a single run with no agent or tool harness. Dai says the evaluation images and text were withheld from training. He says a contamination check found no overlap "in the training data checked."

Elorian also claims Glimpse matches or beats frontier models on two robotics benchmarks, WGO Bench and ERQA, at lower cost. The page gives no numbers for that claim.

Limits

The Glimpse result does not use the same protocol as the paper's models: one run against a three-run average. The paper does not report it, and no one outside Elorian has checked it. The contamination statement covers only the data checked. The page gives no access details, pricing, model size or release terms for Glimpse or the DecoyCount dataset. It also gives no per-error-type breakdown, although separating error types is the benchmark's stated purpose.