[AINews] AEF-1 standard emerges for Third Party Evaluators, as Xai, OpenAI, and Anthropic all cosign

What it claims

The issue makes two kinds of claims. The first is about safety governance. The AI Evaluator Forum, formed in December 2025, published AEF-1, a proposed baseline for independent third-party AI evaluations. It covers five areas: access, conflicts of interest, funding relationships, recusal, and transparency. Dario Amodei's personal follow-up to "Pacing the Frontier" sets out a three-tier pacing plan:

1. Embedded Evaluators. Third-party teams such as METR get ongoing, employee-like access. They verify adherence to safety commitments, report incidents, and assess the alignment of training pipelines and processes, not only finished models. Anthropic says it is committing to this unilaterally now. The concrete access listed is "desks in our offices, access badges, and company laptops" and "access to workspaces, tools, and permissions mostly comparable to what internal risk assessment teams have." The stated precedent is regulatory supervisors embedded in banks.
2. Democratic Coordination. Labs in democratic countries agree common safety standards and limits on the rate of progress. The post concedes that some forms of this are legally challenging and need government support.
3. Global Coordination. Democratic governments attempt coordination with authoritarian ones, "taking seriously the challenges of verifying compliance."

The newsletter identifies the third tier (pacing with China) as the real test.

The second kind is a set of engineering datapoints from the week's news, which the summary names without numbers. These are covered under Evidence.

Evidence

• DeepSeek-V4.1-Flash (Max), Agent Arena:
• Ranked #3 among open models and sits on the cost/performance Pareto frontier.
• Scored +4.87% net improvement at about $0.06–$0.07 median cost per task.
• Comparisons: Hy4 preview at +4.96% / $0.22, and Kimi K3 (Max) at +6.39% / $0.77.
• LangChain: changing the file-reading format cut edit_file errors by 15% and total input tokens by 10%.
• OpenAI desktop voice: price cut by about 60%, followed by a 2.4× increase in usage.
• MiniMax H3: generated 14.4 s of 768p video in 9.0 s end-to-end, after warmup, on 8× B200.
• Cognichip ACI Enterprise: covers spec-to-RTL, verification and PPA optimization. In one reported run, a single engineer finished in 10 days work that Cognichip compares to 4–5 months for a traditional front-end team. This is an anecdote, not a benchmark.

How to use it

• Harness-from-scratch recipe (Omar Shorbagy):
• Separate inference, tools and the agent loop.
• Keep prompts minimal, log aggressively, and test on diverse tasks.
• Only then add memory, skills and subagents.
• Claimed cost and reliability gains come from slimmer prompts, routing, compaction and verifiers.
• GitHub Copilot: auto model selection has three tiers (efficiency, balance, intelligence). There is also a Jira canvas and an /ask mode that works while the agent is running.
• Cline Desktop: a native app with BYOK and provider choice. It supports open-weight models including DeepSeek-V4.1-Flash and Musespark-1.3, and lets you switch models mid-project.
• vLLM on TPU (Inferact + Google Cloud): includes production serving features, optimized kernels, and a native PyTorch path via TorchTPU. A community program offers TPU capacity and maintainer support to open-source contributors.
• Arcee Forge with Bolt: opted-in Bolt Pro users get 50× more usage on open-weight models. In return, anonymized development-session data is used for training and evals. Weights are promised for public release afterward.
• Cohere Parse 5: cost-competitive. Per Jerry Liu, it is weaker than alternatives on visual grounding, chart parsing and fine-grained citation extraction. Check these before switching.
• Tinker: used with physics-based verifiers to train models that design power transformers meeting real-world specs. This is an RLVR template for domains that already have simulators.

Limits

• This is a newsletter digest of tweets. Nearly every number is secondhand, and no methodology is given for the Arena, LangChain or MiniMax figures.
• The page does not reproduce the AEF-1 text. It gives only the five areas the standard covers.
• The body never substantiates the title's claim that xAI, OpenAI and Anthropic cosigned AEF-1. Only Anthropic's unilateral embedded-evaluator commitment is described.
• Dan Selsam, an OpenAI researcher, argues that situationally aware models may game evaluations by recognizing when they are being tested. This undercuts the evidential value of exactly the evaluations AEF-1 standardizes.
• Kevin Bass alleges financial entanglement between Anthropic and parts of the eval ecosystem. That bears directly on AEF-1's conflict-of-interest provisions.
• RewardAI's OM-1 claims (zero-shot transfer across tabletop, industrial and humanoid robots; near-human dexterity) come with no evaluation numbers.