📊 Evals & Benchmarks465

AINews issue for 29 to 30 October 2026 from Latent Space. It opens on Google's admission that AI Mode monetises worse than classic Search, which preceded a roughly 15% share drop. It then recaps Claude Ultra dropping the 5-hour limit, Gemini 4 Carbon, Mistral Large 4, Nemotron 3.5, Artificial Analysis Index v5, Unsloth GGUF disputes, and the Anthropic S-1 filing.

latent.spacearticlePublished and added 31 Oct 2026

The paper treats a benchmark as an empirical measure over item difficulty and formalizes two failure modes: measure mismatch (benchmark difficulty distribution differs from the intended one) and coverage gaps (uncovered difficulty regions that hide localized training progress). It proposes two diagnostic metrics with theoretical guarantees and analyzes 25 popular benchmarks across 7 domains using estimated item difficulties. It also introduces BenchFill, a difficulty-targeted item rewriting algorithm that reduces local coverage gaps.

huggingface.copaperAdded 31 Oct 2026

Hyperflow H3 is Video Rebirth's 8-step LoRA for MiniMax H3, released in September 2026 with open weights. Artificial Analysis ranks it #5 on AA-Video-I2V v2.1 and #3 on AA-Video-Editing v1.1, ahead of MiniMax H3 (768p). It costs $9.00 per minute for Image to Video and $18.00 for Video Editing at 1088p, under half of Dreamina Seedance 2.5 ($34.12 and $40.82). The thread gives per-capability and per-use-case breakdowns.

x.comthreadAdded 31 Oct 2026, published 30 Oct 2026

AINews issue for 28 to 29 October 2026 from Latent Space. It opens with a preview of the AIE CODE summit (themes: Scaling without Slop, System One models, Increasing Ambition, with Jarred Sumner on Bun in Rust). It then recaps Twitter and Reddit news: Mistral Large 4 open weights (1.05T total, 52B active), Artificial Analysis benchmarks, Gemini 4 Carbon, Qwen4 rumours, Claude Code quota complaints and OpenAI frontier-run monitoring. Text was cut before the end.

latent.spacearticlePublished and added 30 Oct 2026

llm-trails is an open study of what LLMs choose to do next inside a breached AWS account. It puts 14 models in 13 post-compromise situations, records only a single-turn plan with nothing executed, and codes each answer to CloudTrail events. The repo has the llmtrials harness, prompts, codebook, processed dataset (viz_data.json) and an interactive dashboard, so the study can be rerun or extended.

github.comrepoAdded 30 Oct 2026

Artificial Analysis launched an HLE-Diamond leaderboard in v5 of its Intelligence Index, covering the refined 1,000-question Humanity's Last Exam from CAIS and Scale AI. Each question runs once without tools, graded by GPT-5.6 Luna (medium) as pass@1. Gemini 4 Carbon (high) leads at 69%, Claude Opus 5.5 (max) scores 68%, and the top open weights model, Kimi K3.1 Pro, reaches 37%.

x.comthreadPublished and added 30 Oct 2026

GPT-Rosalind-6 System Card Addendum describes OpenAI's life-sciences reasoning model, which uses the same weights as GPT-6 Astra and is offered only through trusted access. It reports life-science benchmark gains over GPT-Rosalind-5.5 (MedChemBench v7 49.6% vs 28.9%), High-capability Biological and Chemical Preparedness results, and below-threshold Critical bio design evaluations. It also covers access controls, refusal evaluations, cyber jailbreak regression and alignment tables.

deploymentsafety.openai.compaperPublished and added 30 Oct 2026

dfbench is depthfirst's held-out benchmark for frontier models on open-ended defensive security work: vulnerability detection, validation of findings, and differential analysis between commits. It reports recall, precision, F1 and cost per task for about 20 models at several thinking levels, split by domain (application, low level, smart contract) and by single or multi-repo scope. Overall detection recall ranges from about 21% to 69%.

depthfirst.comarticleAdded 29 Oct 2026

Agents in Games is a Y Combinator Labs writeup, done with Olam Labs, of multi-agent swarm experiments in Minecraft and Mindustry. A 50-agent GPT-6.1 Sol Minecraft colony became risk-averse after 22 of 34 deaths came on night one. GPT beat Claude Opus 5.5 at Bed Wars 6-3 and Mindustry 6-1. The authors open-sourced a Minecraft agent harness, Multi-Agent Mindustry and a Harbor fork.

labs.ycombinator.comwriteupAdded 30 Oct 2026, published 29 Oct 2026

Overtone is a caption format (.overtone.json), a CLI and a video.js plugin that store each spoken word once and cut captions separately per screen shape (16:9, 9:16 or custom). The transcribe pipeline combines gpt-transcribe, Azure AI Speech and optional ElevenLabs Scribe, resolves disagreements with an LLM vote, and uses an LLM editor for corrections. A small public benchmark reports 5.3% word error rate and $0.023–0.025 per audio minute.

github.comrepoAdded 30 Oct 2026, published 29 Oct 2026

VeriCUA is a platform-agnostic framework that verifies task completion by computer-using agents through agentic exploration. The verifier inspects context to build a task rubric, then picks GUI or code checks for unresolved requirements and combines findings under explicit evidence rules. It works online and offline across desktop, mobile and web, and agrees better with reference labels than the baselines, with the largest offline gains on hard cases.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

This paper derives bounds on predicting an LLM's decode length from prefill-phase hidden states, a primitive for inference scheduling. It models decoding as a stopped stochastic process, giving per-prompt normality and an affine variance law σ=αμ+b. Predictors (MLP, EGTP, FiLM, soft-label, Mixture-of-Experts) converge near the bound, so system error dominates model error. It proposes fixing model, data and sampling in benchmarks and reporting relative-error metrics.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

PhysCount is a benchmark of 546 real-world images and 4,595 human-written questions in 651 target-specific groups. It tests whether multimodal LLMs can separate visible appearances (direct view, reflections, glass) from the physical entities behind them. Across 40 MLLMs, counting appearances is much easier than linking them to entities, and the best model answers every question correctly in only 13.36% of groups.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

EpiCoach is an agentic system that treats one cross-country mountain biker as an N-of-1 experiment. Claims live in an evidence-gated hypothesis registry, and each must make a probabilistic prediction before a session. Predictions come from a digital twin fitted to heart rate, GPS and altitude. In a replay over eighteen months the twin beat standard forecasters on heart rate but not segment time, and the agent loop matched a static model. In the first three live weeks, the ledger sealed 20 predictions and scored 14, and a safety officer vetoed the agent twice. The registry, benchmark and protocol are released.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

DecoyCount is a benchmark from Elorian and Mercor with 1,660 items that tests whether models count objects or fall back on visual shortcuts. Frontier models scored below 50%: GPT-6 Astra led with 43.17%, ahead of Gemini-3.8-flash (32.71%), Qwen3.8-max (28.31%) and Claude-Fable-5.1 (25.22%). CEO Andrew Dai says Elorian's Glimpse model scored 50.48% in a single internal run without tools, and contrasts visual reasoning with world models such as Atlas and Genie 4.

latent.spacearticleAdded 28 Oct 2026, published 29 Oct 2026

OpenAI describes its multilayer misalignment monitoring system for frontier models: activation probes, a model-based reviewer, investigative agents, automated log-based investigation and human paging. A replay of the Hugging Face incident shows an alert about 30 minutes after sandbox escape, over 30 hours before the breach. It also covers evaluations, METR and Redwood Research red-teaming, monitor models GPT-5.6 Sol, Terra and Luna, and avoiding pressure on chains of thought.

alignment.openai.comarticleAdded 30 Oct 2026, published 29 Oct 2026

MAServeBench is a benchmark that treats multi-agent collaboration as a configurable dimension of LLM serving workloads. Its composable stages model how agents coordinate and exchange context in documented multi-agent workflows, and Hierarchical Semantic Traces record operations and dependencies for reproducible comparisons. Task-linked measurements tie workloads to GPU/NPU behavior, showing serving efficiency does not always mean timely task completion. Code is released.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

Gemini 4 Carbon is Google's new frontier model, announced with scores of 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench. Google says it beats GPT-6 Astra, Claude Opus 5.5 and Claude Fable 5.1 on these, with no guardrails. Access is limited to Fairwind Program partners at first. Pricing starts at $2 input and $10 output per million tokens, rising to $4 and $20.

securityweek.comarticlePublished and added 29 Oct 2026

A five-task benchmark tests multimodal LLM color judgment against CIE colorimetric ground truth: AWB pair judgment, skin-tone bias direction, CCT estimation, Delta E 2000 pair ordering, and color naming under changing illuminants. Eight endpoints were tested. On ColorChecker-24 stimuli, closed frontier models reach CCT MAE of 715-1056 K versus 1358-2133 K for open-weight models, and a non-ColorChecker control shows this gap is partly template-dependent.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

Mistral Large 4 ("Le Chonk") is an open-weights model from Mistral, now on Hugging Face. Per Mistral's announcement quoted in the post, it ranks #1 among open models on Cognition's FrontierCode benchmark, is aimed at agentic coding and cybersecurity, and claims better reasoning efficiency and lower inference costs. The post does not give parameter counts, license or benchmark scores.

x.comthreadAdded 30 Oct 2026, published 29 Oct 2026

Gemini 4 Carbon scores 59 on Artificial Analysis Intelligence Index v5 (published 28 October 2026), the top score, with #1 on 4 of 9 evaluations including HLE-Diamond and AA-Omniscience. On the Coding Agent Index it scores 59 with Antigravity CLI, 3 points behind Claude Code with Claude Opus 5.5 (62). Cost is $7.76 per task, versus $10.04 for Opus 5.5. Rollout is limited to selected users.

artificialanalysis.aiarticleAdded 30 Oct 2026, published 29 Oct 2026

Google reports two classroom studies of Gemini. In a 10-week RCT with over 1,100 UK students (ages 12-15) on Eedi, Gemini-based AI tutoring matched human tutors on in-platform tasks and quizzes, with expert approval of responses rising from 76% to 95%. Gains did not appear on STAR standardized tests. A yearlong Italian study of 1,600 teachers found lessons co-designed with Gemini shifted toward collaborative work.

blog.googlearticleAdded 30 Oct 2026, published 29 Oct 2026

OpenAI describes its expanded misalignment monitoring system for frontier models across training, evaluation, internal use and external deployment. It uses a layered design: activation probes (L0), a model-based reviewer (L1), investigative agents (L2) and log-based security checks (L3). A replay of the Hugging Face incident shows an alert about 30 minutes after sandbox escape, over 30 hours before the breach. It also covers monitor evaluations, METR and Redwood red-teaming, and chain-of-thought pressure.

openai.comarticlePublished and added 29 Oct 2026

AINews issue for 27-28 October 2026, a Latent Space digest of Twitter and Reddit. It leads with Google DeepMind's Gemini 4 Carbon: claimed cyber scores (SEC-bench Pro 88.2%, ExploitGym 54.8%, CWE-bench 71.0%), #1 on Artificial Analysis Intelligence Index v5 at 59, and access limited to cyber defenders through the Fairwind Program. It also covers benchmark integrity disputes, GitHub Copilot Fusion, H-Net v2, KV-cache work, open-weight releases and Claude Code quota complaints.

latent.spacearticlePublished and added 29 Oct 2026

RetroChessBench is a 12-task benchmark for retrograde chess: reconstructing legal game histories, exact-length constructions, unique endings and exhaustive answer sets, with references, rubrics and a terminal-reward RL environment. GPT 6 Astra High fails to give the required final answer in 22 of 24 attempts without tools and 18 of 24 with code execution. Mean terminal RL rewards are 14.75% and 27.19%. The authors link the failures to scarce worked derivations in web training data.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

PrivacyEvalClaw is a tool-augmented deliberative agent for auditing privacy leakage in LLM responses produced under jailbreak prompts. By Krishna Kanth Nakka and five co-authors, the paper is on arXiv (cs.CR) and accepted to AACL-IJCNLP 2026. The fetched post gives only the title, authors and venue, so method details and results are not available here.

x.compaperAdded 30 Oct 2026, published 29 Oct 2026

Project ProtoStar is an initiative by Carnegie Mellon, Purdue, Georgia Tech, Arizona State, George Washington University, Astera Institute, ORCA and Hugging Face to release publicly funded science datasets, models and code in AI-ready formats. Hugging Face reports over 17,000 public science datasets and tripled sharing in 2026. The first challenge, run by CMU from 2 to 13 November 2026, asks teams to forecast climate indicators for 210 US cities.

huggingface.coarticlePublished and added 29 Oct 2026

Mistral Large 4 (Le Chonk) is a multimodal, multilingual Mixture-of-Experts model with about 1T total and 49B active parameters and a 1M-token context, released under the Mistral Ecosystem Open-Weight License. The post covers its MLA and DeepSeek Sparse Attention architecture, a 1.6B Pixtral encoder, and benchmarks including 82% on the Artificial Analysis Cyber Index and 93% on Cybench. It gives run recipes for transformers, vLLM, llama.cpp, TRL fine-tuning and Inference Endpoints.

huggingface.coarticlePublished and added 29 Oct 2026

The addendum to the GPT-6 Luna system card covers the October 29, 2026 update of GPT-6 Luna, which gives more concise answers and less agreeable language. It compares the update with GPT-6 Luna (October 7) on safe completions, under-18 evals, jailbreaks, HealthBench, MentalHealthBench, hallucination and alignment. OpenAI keeps its High rating for cybersecurity and for biological and chemical capability. Hallucination error rates rose, and HealthBench scores fell.

deploymentsafety.openai.comarticlePublished and added 29 Oct 2026

TrussTutor-Bench is a benchmark of 200 verified two-dimensional truss problems from statics, paired with automatically generated student solutions. It evaluates thirteen LLMs on solution accuracy (64.5% to 97.0%), error detection (90.4% to 100%) and sycophancy. Models rejected their own correct solutions in 1.6% to 50.0% of cases when a student insisted the work was wrong. Sycophancy separated models; error detection did not.

arxiv.orgpaperAdded 31 Oct 2026, published 28 Oct 2026