arxiv.orgpaperHard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Vision
This paper systematically evaluates GPT-6 Astra alongside five other frontier general-purpose AI systems across 34 capabilities and 55 benchmarks spanning nine areas of computer vision, benchmarking them against dedicated specialist models and human performance where available. It finds general-purpose systems increasingly match reference performance on semantic interpretation, reasoning, and object-centric prediction, but still lag on metric geometric accuracy, faithful reconstruction, temporally consistent dense prediction, and fine-grained specialist visual knowledge. Astra shows the strongest gains among the systems tested, particularly in visual/spatial reasoning and structured prediction tasks.
What it claims
The paper gives a per-capability scorecard showing where a general-purpose model can replace a dedicated vision model today. GPT-6 Astra matches or beats the reference model on 2D detection, segmentation, 3D visual grounding, text-to-image generation, editing, image quality assessment, and driving and embodied understanding. Every one of the six frontier systems still trails dedicated models by a wide margin on image restoration, multiview reconstruction, pathology, and remote-sensing grounding. The other five systems are Fable 5, Kimi K3, Gemini 3.1 Pro, Qwen 3.8-Max and Muse Spark 1.3. Astra's lead over the next-best generalist is largest on video segmentation (+26.6), 2D pose (+22.7), 3D medical grounding (+27.9) and visual logical reasoning (+13.6).
How it works
- Same inputs for every model. All six systems get identical instructions, visual inputs and samples on each benchmark. Each model runs at its strongest available reasoning setting, and each benchmark uses its own native metric.
- Structured outputs are scored directly. Boxes, masks, depth maps, 3D predictions, mask sequences, generated images and embodied actions are all evaluated, not just text answers.
- "Reference model" is not always a specialist. It is either a dedicated specialist or the prior generalist SOTA, depending on the benchmark.
- Human scores are partial. They come from the benchmarks themselves and are included only where reported.
- Maturity tiers. Results are grouped as exceeds / at / approaching / substantial gap. The authors state these tiers describe benchmark maturity, not whether a task is solved.
Evidence
Selected rows from Table 1 (best generalist vs. reference; ↓ = lower is better):
| Capability | Astra | Reference | Note |
|---|---|---|---|
| 2D object detection | 89.1 | 78.4 | Qwen 3.8-Max also beats it (87.0) |
| Segmentation | 81.3 | 77.0 | others 41.3–70.9 |
| 2D pose estimation | 75.2 | 83.6 | others 19.6–52.5 |
| 3D visual grounding | 73.8 | 46.5 (human 95.0) | all six beat the reference |
| 3D object detection | 17.3 | 16.8 | |
| Depth (↓) | 0.63 | 0.44 | |
| 3D reconstruction (↓) | 1.40 | 0.61 | ~2.3× the specialist's error |
| Video segmentation | 84.5 | 91.0 | |
| Image restoration (PSNR dB) | 17.7 | 30.7 | all models 17.2–17.7 |
| Microscopy & pathology | 23.8 | 57.1 (human 82.0) | |
| Remote-sensing grounding | 26.4 | 75.5 | |
| 2D medical grounding | 61.5 | 81.8 | |
| 3D medical grounding | 73.9 | 59.4 | |
| Doc & chart understanding | 92.8 | human 89.3 | all six above human |
| Visual math reasoning | 92.5 | human 78.7 | all six above human |
| Visual logical reasoning | 81.1 | human 87.0 | others 57.6–67.5 |
| Robotic navigation | 78.0% success | – | no reference |
Two patterns stand out:
- Restoration is a shared ceiling. All models land within 0.5 dB of each other, so switching generalists will not help there.
- The gap is not uniform within an area. In both 2D and medical vision, grounding and detection can be at specialist level while a sibling task, such as pose estimation or 2D medical grounding, still trails by 8–20 points.
How to use it
- Code: GitHub
mbzuai-oryx/frontier-vision. Project page: mbzuai-oryx.github.io/frontier-vision. - The fetched text gives no run commands, licence, or per-benchmark list.
- In practice, Table 1 works as a lookup. Treat rows with a large negative "best vs ref" gap as places where you still need a specialist model. Treat rows at or above the reference as candidates for a general-purpose API.
Limits
- Truncated page. The fetched text cuts off before the results for RQ3 and RQ4. RQ3 covers which task properties predict the remaining gap; RQ4 covers whether reasoning, tool use, or specialist-as-tool pipelines close gaps, and at what cost. The abstract says only that tools and reasoning "close selected gaps" with varying benefit, so no cost or tool-use numbers are available here.
- Prose and table disagree. The prose says the best model is 7.9 points behind on video segmentation, while the table shows 84.5 vs 91.0 (−6.5). The prose's video-understanding range (72.6–76.3 vs 73.3) also does not match the table. Check the full paper before quoting either.
- Model coverage gaps. Fable 5 is marked not supported for generation, editing, restoration and IQA. Kimi K3 is not supported for generation and editing.
- Weaker baselines on some rows. Where the reference is a prior generalist rather than a specialist, "exceeds reference" is a weaker claim.
- Unreported settings. The reasoning setting used for each model, and the sample counts per benchmark, are not given in the fetched text.
Technologies named here
Related in evals
