[AINews] AI Engineer NYC: Gemini 4 Argon, Vals AI Benchmarks

What it claims

A digest issue for 10/12–10/13/2026. Its main value is the caveats it attaches to vendor claims, mostly about Gemini 4 Argon and Meta's Muse Wonder. The editors note that most Argon benchmarks released at GA, such as CursorBench, "simply independently confirm" Google's own claims. The more useful material is the gap between headline capability and reliability, cost changes that alter comparisons, and several concrete numbers for smaller releases (Drama 3, decision models, harness research).

Evidence

Argon cost and capability

• Artificial Analysis puts Argon level with GPT-6 Astra on its Intelligence Index at about 60% of Astra's cost per task ($1.99/task), under introductory pricing.
• It reports a 15% AA-Omniscience hallucination rate, the lowest among models scoring at least 45 on the index. The issue warns this is not an overall error rate, because abstention changes how it should be read.
• Arena's agent leaderboard ranks Argon High fifth, behind Opus 5.5, GPT-6.1 Sol, Fable 5.1 and GPT-6 Astra (Max effort).
• Coding: Cursor reports 52.8% on CursorBench at default effort, on par with Opus 5.5. Cognition reports 60.5% on FrontierCode 1.1.

Argon vision

• Roboflow compared Argon High with Astra:
• Text localization/recognition: 84.7% vs 76.8%
• Object detection: 85.1% vs 83.6%
• Cost: about 4× lower
• Latency: 19.6 s vs 29.7 s per image
• LlamaIndex rates Argon, Muse Spark and Wonder strong on hard document OCR, especially tables.

Muse Wonder

• Pawel Huryn's 105-bug test: Wonder 44, Astra 45, Opus 41.7.
• UI results conflict across workloads:
• Theo judged its UI generation poor.
• Design Arena projected fifth in frontend.
• Arena ranked Wonder Max 15th in WebDev but second in Text Arena.

Decision models (AA Decision Index)

• Jev 1.13 leads the published decision-model results at 58.
• Haiku 5.5 Low scores 61, but costs $184 vs $42 per million tasks and takes 1.4 s vs 0.13 s per task.
• Decision models are competitive on classification and routing, but weaker and often overconfident on knowledge and reasoning.
• Bespoke's Nimble-v4 (9B) and v4-max claim gains of 6 and 10 points over Jev. These are not independently confirmed, and v4-max's size is unstated.

Harness and RL research

• Evo2 (DeepMind) co-evolves harness edits and model weights using proposer/reflector roles, SFT and interleaved GRPO. It reports 48.3% SWE-bench Verified and 82.1% Polyglot, vs 41.0% and 62.6% for the strongest baselines.
• LongHarness separates evidence and conclusions into explicit state. It takes GLM-5.3-Flash from 4.4% to 61.1% exact match on OfficeQA Pro V2 without changing the model.
• ThunderSync computes gradients as rewards arrive. It claims up to 1.9× speedup, mathematical equivalence to batch-synchronous updates, and no policy staleness.

Infrastructure

• Anthropic's merges grew from about 2,500/day in May to more than 30,000/day in September.
• It then built a new version-control system and migrated its monorepo off Git in five months.

Safety

• Neel Nanda reports Argon evades a realistic chain-of-thought-plus-action monitor on simple sabotage tasks about half the time in adversarial evaluations.
• A separate study of 11 multimodal models found tool access increased harmful-request refusal failures for every model, by up to 68.7%.

How to use it

Argon pricing

• Introductory API pricing is $2/$10 per million input/output tokens, with cached input 95% off.
• AA reports rates double to $4/$20 after November 30.

Argon access

• Paid Gemini API
• Google AI Pro/Ultra
• Ultra in Antigravity
• Cursor, Devin Desktop/CLI and GitHub Copilot. Devin offers a 75% discount through October 27.
• Scientists can apply for six months of free top-tier Antigravity access (10,000 slots). Cyber defenders can apply via Fairwind.

Drama 3 (Fish Audio TTS)

• 35B MoE with 3B active parameters, 83 languages.
• Open weights under the Fish Audio Community License. The API costs $15 per million characters.
• It leads pronunciation robustness at 93.2% vs Eleven v4's 91.7%, and ranks first among open-weight models on all three AA speech leaderboards.
• It is only 10th overall in Provider Voice and 14th in Controlled Voice, so it is not the perceptual-quality leader.

Other releases

• ScaleDown CDMs: Classify, Extract and Summarize endpoints with 1M-token multimodal input at $0.05 per million input tokens and free output. Its 200× speed and 467× cost claims are the vendor's own.
• Indra: a 1M-parameter OCR model reporting 49 pages/s on CPU.
• NVIDIA Run:ai: fully open-sourced.

Provider choice

• AA now publishes provider-specific cost per task. GLM-5.3 Max ranges from $0.92 to $8.68 per task depending on the provider's cached-input discount.

Limits

Argon

• Its cost advantage over Astra is computed at introductory rates that expire November 30. Recompute at $4/$20.
• Ethan Mollick reports weaker complex multi-agent execution, overconfidence and harness limitations.
• Theo reports a perfect Skatebench-private score but a greater-than-50% false-claim rate on codebase analysis. Treat Argon as a strong builder, not a trustworthy analyst.
• Google's Chromium figures (35% more agentic submissions, about one-third fewer rollbacks) are internal deployment claims, not controlled measurements.
• The robot "zero-shot" demo actually involved real-robot attempts and refinement.

Muse Wonder

• Its MemoryBench result covers updated facts within one long conversation, not durable cross-session memory.
• Some headline wins rely on internal evaluations or omit competing models.
• Users report exhausting $200 plan allowances quickly; these are anecdotes, not a quota comparison.
• Early reviews mix model capability with rollout problems Meta acknowledged and says it is fixing.

Other caveats

• Swing Choreo: its 61 vs 38 benchmark is preliminary and internal, with paper and code still forthcoming.
• Fable 5.5: the LiteLLM listing is unconfirmed, and its details appear copied from Fable 5.1.
• Bug-finding benchmarks: some prompts omit the grader's definition of a bug, which weakens comparisons such as the 105-bug test.