Skip to content

32 results for “Calibration”

ListGrid
What you seeeverything
RecentOldestAddedA–Z
All types32Papers26Articles3Repos2Threads1
“Calibration”
arxiv.orgpaper🧠 AI

M-PFT: Angular Margin for Calibrated Prompt-Tuning of Fine-Tuned Vision-Language Models

M-PFT is an angular-margin method for prompt tuning of vision-language models that fixes base-to-novel dual miscalibration (underconfidence on base classes, overconfidence on novel classes). It adds two losses: minimizing the maximum cosine similarity between tunable class embeddings, and minimizing angular distance to each zero-shot embedding. Tests on 9 prompt-tuning methods and 15 datasets show lower novel-class calibration error at comparable accuracy and negligible overhead.

vision-language modelprompt tuning
arXiv

research, multimodal, training· Added 31 Oct 2026, published 29 Oct 2026

huggingface.coarticle🧠 AI

Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms

AnySpeechRecognizer and AnyDecisionModel are two Swift packages from Hugging Face. The first mirrors Apple's SpeechAnalyzer API with swappable engines (NVIDIA Parakeet, Alibaba Qwen3-ASR on Core ML) and adds Sortformer speaker diarization. The second returns typed yes/no, choice and score answers with probabilities via MLX, Core AI, llama.cpp, Jev or Ollama. The post gives M4 Max timings, model sizes and calibration options.

SwiftCore MLMLX+10
AnySpeechRecognizerAnyDecisionModelAnyLanguageModel+17

multimodal, inference, models· Published and added 29 Oct 2026

arxiv.orgpaper🛡️ Security

Session-Bound Behavioral Watermarking for Autonomous LLM Agents

AgentAnchor is a session-keyed behavioral watermark for attributing LLM agent trajectory logs to an owner. It derives per-step keys from a secret key, a fresh public session ID and semantic observation states, and embeds parity evidence through a KL-bounded exponential tilt. The paper formalizes learned cross-session forgery, and a trusted verifier checks disputed logs using wrong-key calibration and abstention.

LLM
AgentAnchorlearned forgery

AI Security· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper🧠 AI

Session-Bound Behavioral Watermarking for Autonomous LLM Agents

AgentAnchor is a session-keyed behavioral watermark that lets an agent owner prove ownership of a trajectory log. Each step key comes from a secret key, a fresh public session identifier and semantic states of recent observations. Parity evidence is embedded through a KL-bounded exponential tilt of action choices. A trusted verifier checks disputed logs without rerunning the agent, using wrong-key calibration and abstention. The paper also formulates a learned cross-session forgery threat.

LLM
AgentAnchorarXiv

research, agents· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper🧠 AI

Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants

This paper builds a controlled French surface-variant benchmark with seven task types and 4,200 original-low-high triplets (12,600 inputs), covering missing apostrophes and diacritics and keyboard typos. Six models from two LLM families lose accuracy and show worse Brier score and NLL as perturbation grows. Keyboard typos hurt most. Calibration temperatures fitted on original inputs transfer to variants, and combining them with input restoration improves reliability further.

LLM
arXiv

evals, research· Added 27 Oct 2026, published 24 Oct 2026

arxiv.orgpaper🧠 AI

Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

This paper studies reverse sycophancy: whether users bend toward a confident, consistent, wrong LLM. In a within-subjects experiment, 32 participants learned science topics from five GPT-5-mini assistants with 0 to 100% of facts replaced by falsehoods. Scores fell from 7.4 to 2.9 out of 10, planted falsehoods were chosen on 60% of falsified items against 11% of correctly taught ones, and confidence stopped tracking correctness. It introduces a baseline-corrected Human Deference Index.

LLM
GPT-5-miniarXiv

research, ai-safety, evals· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

Beyond 4 bits: Data-Centric Post-training Quantization and Recovery

This paper studies 2-bit post-training quantization (PTQ) of six dense and mixture-of-experts models from 3.8B to 32B parameters. It finds the quantization representation matters most: vector codebooks give usable models where scalar PTQ falls to chance. Calibration and recovery data must be chosen jointly. The recipe uses about 2.3 effective bits per weight, keeps 89 to 97% of full-precision QA accuracy, and needs 1.4 to 30x less compute than the strongest QAT baseline.

Mixture of ExpertsLLM
arXiv

research, inference· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper🧠 AI

SHADE: Shrinkage via Halted Alpha-stable Denoising for Heavy-Tailed Probabilistic Time Series Forecasting

SHADE is an alpha-stable Lévy diffusion forecaster that stops denoising at a nonzero depth, so the leftover stable kernel shrinks the predictive distribution instead of leaving it overconfident. It offers a scale-calibrated halt and a context-amortized halt that needs no noise labels. Median CRPS improves 21.7% on synthetic data with alpha at or below 1.3, with tail calibration gains on cryptocurrency and foreign-exchange series.

diffusion model
arXiv

research· Added 21 Oct 2026, published 19 Oct 2026

arxiv.orgpaper🧠 AI

AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

AI Consilium is a multidimensional benchmark built from 300 diagnostically resolved NEJM–MedQA cases (110 for physician reference ratings and Judge calibration, 190 held out). Eight single-model pipelines produced role-specific opinions and interactive-consilium reports under free-form and structured clinical-reasoning conditions, and a physician-calibrated LLM Judge scored 3,040 final reports. Model identity separated quality more than format; structured reports won in seven of eight pipelines (management 2.740 vs. 2.436), multidisciplinary synthesis beat the primary-role opinion on all five shared criteria, and GPT-5.6-Sol led every component with the fewest review rounds.

LLM
AI ConsiliumNEJMMedQA+2

evals, research· Added 17 Oct 2026, published 15 Oct 2026

arxiv.orgpaper🛡️ Security

From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection

Evaluates a native probabilistic decision model (Jev), Llama Prompt Guard 2 (22M and 86M) and an LLM judge on 47,611 held-out examples covering direct/indirect prompt injection, hard-benign inputs, obfuscation and distribution shift. The LLM judge has the best AUROC (0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757) and 22M (0.714). Jev's native probabilities do not consistently beat post-hoc calibrated classifier scores, and temperature scaling improves calibration but not discrimination. At a 1% false-negative-rate target, every system needs over 91.8% escalation, so probabilistic outputs alone do not give reliable security decisions under distribution shift.

LlamaLLM
JevPrompt Guard 2prompt injection+1

AI Security· Added 17 Oct 2026, published 14 Oct 2026

arxiv.orgpaper🧠 AI

Geometric Calibration of Supervised Fine-Tuning

Proposes Geometric Calibration of supervised fine-tuning and SPACE (Subspace Projection for Adapted Checkpoint Enhancement). SPACE is a post-hoc, data-free, training-free method that uses the pre- and post-SFT checkpoints to revert SFT update components lying in the principal subspace of the pre-SFT weights and keep those in the orthogonal complement. It is evaluated on Qwen2.5, Qwen3, Qwen3.5, Llama-3.1 and Olmo-3 models (2B–14B) with math, coding and medical SFT. It reduces out-of-domain regression and improves in-domain performance, for example +2.85 ID / +2.26 OOD on Qwen2.5-7B math SFT.

QwenQwen3Qwen3.5+1
Qwen2.5Olmo-3Qwen2.5-7B+2

research, training· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper🧠 AI

DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

DualQuant is a calibration-free post-training quantization framework for LLM weights and activations. It minimizes weight-reconstruction MSE using variable splitting and block coordinate descent with closed-form subproblem updates. The resulting scaling factors double as per-channel activation rescalings, so no calibration data is needed. It supports the MX family and NVFP4 block formats. On Llama and Qwen models it matches or beats calibration-based SmoothQuant on WikiText-2, C4 and zero-shot tasks. A real packed W4A4 path for Llama-3.1-8B on an RTX 4090 gives up to 2.61x memory reduction and 2.46x throughput over BF16.

LlamaQwenNVFP4+3
DualQuantLlama-3.1-8BWikiText-2+4

inference, research· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper🧠 AI

DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training

DreamTrue is a multi-view, cross-embodiment robot world model for action-faithful, physically plausible video prediction. It renders action trajectories into image-space conditions aligned to target videos by offline geometric calibration. Counterfactual post-training modifies recorded trajectories to cover unsuccessful interactions and contact configurations. A human-annotated defect dataset trains an embodied video reward model whose scores drive RL post-training. On AgiBot it reaches state-of-the-art action following and cuts the human-assessed interaction defect rate from 48.12% to 6.25%. It ranked first in the world model track of the AgiBot World Challenge 2026.

DreamTrueAgiBotAgiBot World Challenge 2026

research, multimodal, training· Added 9 Oct 2026, published 8 Oct 2026

arxiv.orgpaper🧠 AI

TR-PTQ: High-Accuracy Integer-Only Transformer Post Training Quantization via Taylor Region Reformulation

TR-PTQ is a post-training quantization method for transformers that identifies learned LayerNorm scale parameters and GELU's compounded approximations (not SoftMax) as the main sources of accuracy loss when quantizing, contrary to common assumptions about numerical precision. It introduces a unified integer-only formulation using shared Taylor Region exponential/logarithm primitives so division and square roots can run entirely in the log-domain with standard integer arithmetic, paired with a calibration-free, outlier-aware LayerNorm optimization, eliminating the need for floating-point hardware for nonlinearities while keeping accuracy degradation under 1.5% across vision and language benchmarks.

Transformer

inference· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper🧠 AI

Test-Time Adaptation of Quantized ViTs via Single-Pass Quantizer-Aligned Recalibration

This paper introduces Quantizer-Aligned Recalibration (QuAR), a single-pass, backprop-free test-time adaptation method for quantized vision transformers that recalibrates activations at frozen quantizers to counter distribution shift without updating model parameters. On ImageNet-C with ViT-B across 3-, 4-, 6- and 8-bit weight/activation precision, QuAR beats the strongest backprop-free baselines by up to 4.00 points while cutting latency 46% and adding only 0.17 MB memory overhead, with gains holding across continual streams, label shift, seven OOD suites, and three other backbones.

Vision TransformerTransformer
ViT-BImageNet-C

research, inference· Added 7 Oct 2026, published 6 Oct 2026

x.comthread🧠 AI

brrrock (@bchap1n) on X

Announcement of an updated model card for JEV-27B-VL, an open-weight vision-language model released under Apache-2.0. The card covers the model's weights and a "decision adapter," a BoE (Bag/Blend of Experts) architecture paired with a System 1 / System 2 reasoning setup, calibration details and decision prompts, serving instructions for vLLM and SGLang, and an evaluation of image recommendation quality.

vLLMSGLangvision-language model
JEV-27B-VLJev

models, multimodal· Published and added 2 Oct 2026

blog.cloudflare.comarticle🧠 AI

Clef / Clef-flash decision models (Cloudflare)

Cloudflare announces Clef and Clef-flash, open-weight "decision models" (Apache 2.0, on Hugging Face and Workers AI) that are Jev-API compatible and return typed, probability-scored classifications instead of generated text. The models are built by post-training a frozen Qwen backbone (Qwen3.8-27B for Clef, a smaller Qwen3.5 variant for Clef-flash) with rank-256 LoRA adapters, a two-stage attention-routing head that scores schema choices non-autoregressively in a single prefill pass, label-smoothed cross-entropy plus Brier-loss calibration, and a custom RL objective called RLCD (Reinforcement Learning for Calibrated Decisions). Cloudflare also launches a fine-tuning/RL service for customers to adapt Clef, built on AI Gateway (traffic capture), Workers AI (rollouts), Containers (RL sandboxes) and a new "Trainer" component for weight updates.

Hugging FaceQwen3.8-27BQwen3.5+3
ClefClef-flashJev+6

models, training, inference· Added 3 Oct 2026, published 1 Oct 2026

arxiv.orgpaper🧠 AI

Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning

The paper shows that unlearning a fact in one language can leave it recoverable in others, because changing the query language or the answer language reopens the knowledge. It defines language-budgeted multilingual unlearning, where a subset of languages is chosen to maximise cross-lingual erasure. It also introduces the Cross-Lingual Unlearning Tensor benchmark, which covers 174 language–script pairs and 25 atomic paraphrase types. Its COVER method selects source languages to maximise predicted coverage of languages that get no forget supervision, and needs only benign calibration data and the frozen model. Across three model families and two forget sets, COVER cuts mean held-out residual access by 7.8–27.3% versus uniform source selection. The gains carry over to real news documents in low-resource languages from the LORELEI corpus.

Cross-Lingual Unlearning TensorCOVERLORELEI

research, evals· Added 1 Oct 2026, published 30 Sep 2026

arxiv.orgpaper🧠 AI

Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs

This paper introduces Divergent Token Confidence (DTC), a method for uncertainty quantification in LLM chain-of-thought reasoning that counts tokens where two models strongly disagree during decoding, measured via Jensen-Shannon divergence between next-token distributions on the same reasoning trajectory. Unlike standard probability-based confidence scores, DTC requires no training, works in both white-box and black-box settings, and its divergent-token count is shown to be strongly negatively correlated with answer accuracy. Across six mathematical benchmarks and multiple model families (including DeepSeek-V3.2), DTC cuts expected calibration error roughly in half compared to full-sequence probability and verbalized-confidence baselines (e.g., 13.0% vs 32.7%-42.4% white-box).

DeepSeek-V3.2

evals, research· Added 30 Sep 2026, published 29 Sep 2026

magazine.sebastianraschka.comarticle🧠 AI

Language Models for Text Classification: From Bag-of-Words to Jev

Sebastian Raschka surveys the history of text classification — bag-of-words with naive Bayes/logistic regression, RNNs and CNNs with word embeddings, then transformer approaches (BERT/ModernBERT, GPT-style decoder fine-tuning, T5 encoder-decoder) — before analyzing TypeSafe AI's new Jev model, a fast/cheap general-purpose classifier accessed via Choice/Noul/Score APIs. He benchmarks Jev against a fine-tuned ModernBERT on the IMDb sentiment dataset (Jev ~96.2-96.5% vs ModernBERT ~95%), explains how to retrofit a Jev-like classification head onto BERT/GPT models, and discusses probability calibration, including the RLCR (Reinforcement Learning with Calibration Rewards) technique from a 2025 paper that likely inspired Jev's proprietary "RLCD" training method.

TransformerBERTGPT+5
JevTypeSafeGPT-5.6 Luna+15

models, training· Published and added 29 Sep 2026

arxiv.orgpaper🧠 AI

Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory

This paper proposes Rubric-Calibrated Preferences (RCP), a method that combines listwise Bradley-Terry tournaments (relative LLM judgments within a query) with yes/no rubric criteria (an absolute cross-query standard), then uses Item Response Theory to put all queries' tournament scores on one shared scale. The resulting calibrated relevance probabilities feed a new metric, RCP-nDCG, which the authors show correlates much better with human grades (0.538 to 0.795) and separates reranker pairs more reliably than standard nDCG on benchmarks like TREC-DL and NanoBEIR. It matters because it addresses a core weakness in reranker evaluation: nDCG's dependence on sparse, noisy, coarsely-graded human labels that increasingly fail to distinguish close-quality rerankers.

TREC-DLNanoBEIRFabian David Schmidt

evals· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper🧠 AI

A Unified Uncertainty Representation for Graph Neural Networks via Doubly-Spectral Stochastic Expansion

Proposes a doubly-spectral stochastic (DSS) expansion for graph neural networks that unifies calibration, out-of-distribution detection, and distribution-shift robustness in a single representation: graph Fourier filters capture structural variation while an orthogonal-polynomial chaos coordinate captures latent stochastic variation. DSS-GNN (standalone) achieves the lowest Brier score across 14 node classification benchmarks without post-hoc correction, and DSS-Hybrid (a residual branch beside a deterministic encoder) achieves the best AUROC on most node-OOD settings and strongest shifted accuracy on all 7 GOOD concept-shift benchmarks under standard ERM. A capacity theorem shows the method matches chaos coefficients of Gaussian-latent random graph signals under full-rank assumptions with exponentially decaying truncation error.

GOOD

research· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper🧠 AI

Many Eyes, One World: Feed-Forward 3D Reconstruction from Mixed Cameras

MEOW is a feed-forward 3D reconstruction model that jointly predicts metric pointmaps and camera poses from N-view image tuples mixing perspective, fisheye, and full 360° panoramic images in a single forward pass, without any camera calibration, distortion parameters, or camera-type labels supplied. It keeps a perspective-pretrained backbone and adapts it to heterogeneous cameras purely through a procedural synthetic data engine that renders scenes across a continuous manifold of camera models with exact rays, depth, and verified covisibility, then transfers zero-shot to real captures, beating the Wid3R baseline (79.9 vs 53.8 mAA@30 on heterogeneous 2D3DS tuples even when Wid3R is given ground-truth camera types) and scoring 79.4 AUC@30 on a new laser-scanned mixed-camera benchmark.

MEOWWid3R2D3DS

research, multimodal· Added 29 Sep 2026, published 28 Sep 2026

arxiv.orgpaper🧠 AI

SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG

SCAR is an adaptive retrieval policy for RAG that combats boundary fragmentation from fixed-length chunking by selectively expanding neighboring chunks, using a relative expansion threshold tied to each chunk's own query-relevance rather than a fixed window size. Tested on four corpora (RFC, GDPR, a 10-K filing, a merger agreement; 320 queries) it hits 92.8% recall on boundary-fragmented queries using 22.9% fewer chunks than static windowing, transfers across three embedding models (text-embedding-3-large, BGE-large-en-v1.5, zembed-1) without recalibration, and cuts context tokens by 27.1% while preserving RAGAS-measured generation faithfulness.

RAG
RFC corpusGDPRRAGAS+3

llm-apps, research· Added 26 Sep 2026

arxiv.orgpaper🧠 AI

Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software

CWE-Trace is a benchmark and diagnostic framework of 834 manually curated Linux kernel samples spanning 74 CWEs, built with a strict temporal train/test split to rule out data contamination, used to evaluate eight vanilla LLMs and 15 LoRA fine-tuned variants on vulnerability detection and CWE classification. The paper introduces two new metrics — Directional Failure Index (DFI) and Hierarchical Distance and Direction (HDD) — and finds that fine-tuning only recalibrates output thresholds rather than improving actual security reasoning, with the best model reaching just 52.1% detection accuracy (barely above chance) and under 1.3% Top-1 CWE classification accuracy.

LoRA
CWE-TraceDeepSeek-R1Directional Failure Index+2

evals, research· Added 26 Sep 2026

arxiv.orgpaper🛡️ Security

Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software

This paper introduces CWE-Trace, a benchmark of 834 manually curated Linux kernel vulnerability samples spanning 74 CWEs with a strict temporal split to eliminate training data leakage. Evaluating eight vanilla LLMs including DeepSeek-R1 and fifteen LoRA fine tuned variants, the authors find contamination gives no real advantage and that fine tuning only shifts output calibration rather than underlying security reasoning, with the best model barely beating chance and CWE ranking accuracy staying very low. Two new diagnostic metrics are introduced to expose this calibration without comprehension failure mode.

LLMDeepSeekLinux kernel
CWE-TraceDirectional Failure IndexHierarchical Distance and Direction

AI Security, AppSec· Added 26 Sep 2026

arxiv.orgpaper🧠 AI

Two Conformal Constructions for Adaptive Within-Document AI-Text Screening

This paper presents two conformal-prediction constructions for calibrating AI-text detection pipelines that screen document prefixes with multiple detectors and may stop early. Construction A allocates a false-alert budget across conformal ranks for a registered finite family of prefix-detector scores using a union bound; Construction B calibrates the complete-path maximum of an adaptive stopping policy so a single terminal conformal rank protects early stopping without splitting the error budget. Under document-level exchangeability (with no restriction on within-document token dependence), the authors prove marginal false-alert control across the inspection path and derive necessary calibration sample sizes, while noting that detection power and computational savings are left to future empirical evaluation.

research, evals· Added 28 Sep 2026, published 25 Sep 2026

arxiv.orgpaper🧠 AI

Embedding Models Measure in Peculiar Ways

This paper studies whether embedding models' notions of semantic similarity align with objective physical measurements of mass, distance, time and volume. The authors find that physical measurement is only weakly captured in embedding space, and that embedding similarity for quantities is instead heavily driven by superficial string similarity (e.g., shared digits or units) rather than true magnitude proximity, with recalibration attempts failing to fix the misalignment.

evals, research· Added 22 Sep 2026, published 17 Sep 2026

github.comrepo🧠 AI

Mapika/decider

Decider is an open reproduction of TypeSafe AI's System One (Jev) model class: instead of generating text, it reads a state plus a set of typed questions (Choice, Score, or Noul/yes-probability) and returns a calibrated probability distribution for each in one forward pass, with no decoding or parsing. The repo ships 2B/4B/12B and a 35B mixture-of-experts checkpoint (built on Qwen3.5 and Gemma-4 base models), trained on roughly 95 public datasets plus teacher-labelled data, with per-type temperature calibration, GGUF/llama.cpp and vLLM serving paths, an MPS/Apple Silicon port, and a community Snapdragon X Elite NPU port. It benchmarks against TypeSafe's Jev on JevBench and the Decision Index, documenting where it closes the gap (language, tools, retrieval) and where it lags (knowledge-heavy MMLU/GSM8K/GPQA tasks and calibration on hard items).

PythonCUDAMetal+9
TypeSafe AIJevMark Marosi+14

models, inference, training· Added 2 Oct 2026, published 16 Sep 2026

github.comrepo🛡️ Security

y0usaf/pi-jev

pi-jev is an npm extension (@y0usaf/pi-jev) that adds a typed decision layer to the Pi coding agent: a gate scores pending bash/write/edit tool calls on destructiveness, exfiltration risk, scope, and impact before they run, while an output judge scans bash results for leaked secrets and classifies command failures (retry/environment/code_bug/permission/user_error). Answers come back as probabilities or choice distributions rather than prose, are batched into one API round trip to api.typesafe.ai, cached per input, and fail open on any error so a dead endpoint never blocks the agent. The README documents calibration work behind the thresholds (e.g. destructive=0.90, leak=0.90) derived from repeated fixture runs, and details argument/output truncation and key redaction to limit what leaves the machine.

npmTypeScriptNode.js
TypeSafe

AI Security· Added 26 Sep 2026, published 16 Sep 2026

arxiv.orgpaper🧠 AI

SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators

SyncWorld is an action-conditioned world model that acts as a zero-shot simulator across unseen robotic environments without additional training. It uses a "visual calibration episode" — paired frames and actions demonstrating controllable degrees of freedom — to specify a setup-specific Action-Visual Mapping in context, addressing the problem that identical numerical actions manifest differently across cameras, embodiments, and environments. Experiments show it can simulate accurate rollouts in novel settings and enable test-time policy improvement without further training.

SyncWorld

research, multimodal· Added 22 Sep 2026, published 8 Sep 2026

arxiv.orgpaper🧠 AI

Principia: Relational Physics Tests for Video Models

Principia is a benchmark that evaluates Newtonian physics understanding in video generation models using a calibration-independent relational consistency score, testing paired-object motion relationships across eight physical phenomena (gravity, restitution, friction, rotational inertia, projectile motion, momentum, pendulum, mass-spring oscillation). Testing six state-of-the-art video generators shows none exceed 0.42 on Principia despite scoring ~0.8 on VBench, and the best vision-language model achieves only 67% accuracy detecting physics violations, exposing a major gap between visual quality and physical plausibility in generated video.

PrincipiaVBench

evals, multimodal, research· Added 22 Sep 2026, published 3 Sep 2026