119 results for “METR”

The paper treats a benchmark as an empirical measure over item difficulty and formalizes two failure modes: measure mismatch (benchmark difficulty distribution differs from the intended one) and coverage gaps (uncovered difficulty regions that hide localized training progress). It proposes two diagnostic metrics with theoretical guarantees and analyzes 25 popular benchmarks across 7 domains using estimated item difficulties. It also introduces BenchFill, a difficulty-targeted item rewriting algorithm that reduces local coverage gaps.

huggingface.copaperAdded 31 Oct 2026🧠 AI

This paper derives bounds on predicting an LLM's decode length from prefill-phase hidden states, a primitive for inference scheduling. It models decoding as a stopped stochastic process, giving per-prompt normality and an affine variance law σ=αμ+b. Predictors (MLP, EGTP, FiLM, soft-label, Mixture-of-Experts) converge near the bound, so system error dominates model error. It proposes fixing model, data and sampling in benchmarks and reporting relative-error metrics.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026🧠 AI

A five-task benchmark tests multimodal LLM color judgment against CIE colorimetric ground truth: AWB pair judgment, skin-tone bias direction, CCT estimation, Delta E 2000 pair ordering, and color naming under changing illuminants. Eight endpoints were tested. On ColorChecker-24 stimuli, closed frontier models reach CCT MAE of 715-1056 K versus 1358-2133 K for open-weight models, and a non-ColorChecker control shows this gap is partly template-dependent.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026🧠 AI

DeoGen is an automatic, parametrizable benchmark generator for deontic reasoning, where a model applies policy rules to facts to reach a verdict. It autoformalizes rules into first-order logic, builds decision trees, and turns each tree path into a benchmark entry. Frontier models give the correct verdict 89-92% of the time, but only 22-40% on entries lacking enough information, even when told to abstain.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026🧠 AI

AskStream is a multi-turn benchmark for streaming video understanding that adds a third proactive action, ask, alongside wait and answer, for cases where video or user information is incomplete. It has 438 videos, 2,714 manually annotated frame-level time windows (wait, ask, answer), four task categories and eight subtasks. The strongest model reaches only 52.18% on the proposed Ask-F1 metric, in both multiple-choice and open-ended settings.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026🧠 AI

LLMON (LLM Object Notation) is a framework that separates instructions from data and requires explicit identifier-based execution. The authors post-train Granite-4.1 3B and 8B base models with SFT then RL on a 4.6B-token structured corpus covering general, RAG and tool-calling data, with distractor variants. Attack success rates on indirect prompt-injection benchmarks fall from 53-100% to 0.02-5.6%, and general capability is preserved. Attention-based metrics diagnose failures per transformer layer.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026🛡️ Security

The paper proposes POB-ICM (contingency matrix with Chi-square independence testing) and POB-UNBP (a uniformity null model under permutation-based evaluation) to detect positional bias in top-1, single-turn LLM recommenders. On a RAG product-recommendation testbed with 2000 queries and 20 permutations each (40,000 instances), both methods reject positional independence in all evaluated models. Three metrics quantify the effect, and the bias persists across candidate-set size and temperature. Code is public.

arxiv.orgpaperAdded 30 Oct 2026, published 27 Oct 2026🧠 AI

"When Algebraic Structure Meets Too Many Hints" is an ePrint paper (2026/2654) arguing that DDGR hint classes and too-many-hints thresholds are not standalone side-channel security metrics. In synthetic Kyber-512 experiments, NTT-domain residues recover the full secret with about half the hints the uniform mod-q bound predicts. Finite-precision FFT samples need 25% to 55% of the budget for uniform perfect hints. No new attack optimizations are proposed.

eprint.iacr.orgpaperPublished and added 27 Oct 2026🛡️ Security

The Policies of Programming Language Ecosystems is an October 2026 arXiv paper (cs.SE) that compares the policies of eight programming language ecosystems, including CRAN for R, across 38 policy categories. Three authors coded the policies with moderate agreement. Malware, intellectual property and content removal are covered everywhere, while package size limits, metric manipulation, typosquatting and binary uploads diverge.

arxiv.orgpaperAdded 28 Oct 2026, published 25 Oct 2026🛡️ Security

Sentiment-Injected Sentence Embeddings (SISE) addresses a finding across eleven frozen text encoders (2021 to 2026): linear probes recover sentence polarity at up to 91.3% accuracy, but cosine similarity separates polarity by only 0.003-0.064. A small supervised-contrastive head (0.13M-0.69M parameters) trained on CPU raises retrieval MAP by up to 26 points, but the 128-d head loses 38-41 STS-B Spearman points. The paper also releases the Attribute Alignment Gap diagnostic.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026🧠 AI

Sensorimotor Alignment (S^2) is a training-free offline metric that measures how well an embodied model's sensory representation aligns with the geometry of physical motion. It extends the Platonic Representation Hypothesis to motion trajectories. S^2 correlates positively with downstream robot task success across embodied model families, helps pick promising VLM backbones before fine-tuning into VLAs, and explains behaviours seen in VLAs and world action models.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026🧠 AI

F2G-Pose is a direct, single-pass framework that predicts category-level object pose, metric size and camera-frame completed shape from one RGB-D image. It lifts frozen RADIO foundation features onto partial point clouds, tokenizes local geometry with DGCNN, and fuses it with a Mixture-of-Experts Transformer. It trains only on synthetic SOPE data, leads on SOPE and ROPE at 28 FPS, and is tested on HouseCat6D, HANDAL and real manipulation.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026🧠 AI

This empirical study benchmarks recent LLMs against fine-tuned encoder-decoder models on knowledge graph-to-text generation, using reference-based metrics and graph-grounded, reference-free LLM-based evaluation validated against human annotations. Recent LLMs narrow or close the gap on reference-based metrics and do better on graph-grounded ones. Prompted models make fewer contradictions and triple omissions, and fine-tuned models' errors grow faster with graph size.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026🧠 AI

The Geometry of Metaphor is an October 2026 arXiv preprint that uses mechanistic interpretability to find causal subspaces for metaphor in LLMs of different architectures and sizes. The authors steer output on classification tasks across parts of speech. They also steer generation tasks, showing the metaphoricity of model output can be controlled. The abstract gives no model names or metrics.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026🧠 AI

This paper studies evidence-grounded health summarization for menstrual health monitoring. It combines structured evidence conditioning with RAG in a prompt-based constrained generation framework, and scores 4,500 summaries from three LLMs with NLI, atomic factuality metrics and readability measures. Grounding constraints improved uncertainty preservation and generally kept or improved faithfulness against retrieved evidence, while omission effects varied by model. Clinical alignment came with harder reading and more medical terminology.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026🧠 AI

SBFS-FT (symmetry-based few-shot fine-tuning) adapts the MatterSim 1M universal machine-learned interatomic potential with low-rank adaptation on a small set of compounds sharing one crystal symmetry. Tested on half-Heusler (space group 216), space group 225 and spinel (227) datasets, it cut lattice thermal conductivity error 2-3 times versus the base model. It matched per-compound fine-tuning while taking 5-7 times less compute for 100 compounds.

arxiv.orgpaperAdded 23 Oct 2026, published 21 Oct 2026🧠 AI

Code2Video-Compass is a benchmark of 2,104 tasks for code-based video generation, where multimodal LLMs write executable programs that produce complete videos from text (T2V), image (I2V) or video (V2V) inputs. Tasks span 10 categories and 24 subtypes. Scoring uses 12 objective metrics and five MLLM-judge metrics. Across 15 proprietary and open-weight models from nine families, performance falls as construction, dynamic animation and state-dependency complexity rise.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026🧠 AI

DeclineCraft is a dynamic RAG abstention benchmark whose unanswerable questions target Minecraft feature additions made after a corpus cut-off, with audits for parametric exposure. Across 27 LLMs, single-shot retrieval raises Correct Abstention Rate in 38 of 40 configurations, while iterative RAG lowers it in 30 of 40. Enabling thinking without retrieval lowers abstention for 8 of 13 models, shifting responses to premise denial.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026🧠 AI

This paper shows that reward misspecification alone degrades chain-of-thought (CoT) monitorability in language models. Using a testbed of paired true and misspecified rewards across three models and three tasks, CoTs keep reasoning about task intent while outputs hack the reward. The proposed reward description training injects an LM-inferred reward description into the input and raises monitorability by an average of 24% or 42% absolute, depending on the metric.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026🧠 AI

The paper introduces PhysRSI-eval, a five-axis metric for physical recursive self-improvement: first-attempt performance, efficiency, success, budget and improvement from failure. Several VLMs are tested on 18 manipulation tasks under a fixed interaction budget. It proposes the Physical Barrel Hypothesis and studies observational scaling. Workspace interventions show that removing history lowers late-round success, while frozen workspaces still help on unseen layouts.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026🧠 AI

MemT-1.0 is a memory foundation model architecture that builds memory into model computation. It builds on Metis, uses Gated DeltaNet-2 for token-level memory updates within each turn, and adds Mixture-of-Storage to scale capacity. Training used 34B tokens in continued pre-training and post-training. MemT-1.0-Nano and MemT-1.0-Flash beat parametric memory baselines and stay competitive with retrieval methods. At 128K history, Nano gives about 1.86x inference speedup and 66x less persistent storage than a full-context KV cache.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026🧠 AI

"How employment is changing at Anthropic" is an October 2026 technical report by Nathan Wilmers, Regina Catipon, Maxim Massenkoff and Peter McCrory. It uses internal employment, agent telemetry, output, survey, calendar and hiring data plus 17 interviews. Junior technical hiring fell from 27% to 9%, agent hours reached six times employee hours by August 2026, and adopters' output rose. Employees increasingly do meta-work managing agents.

anthropic.compaperAdded 19 Oct 2026, published 18 Oct 2026🧠 AI

Predictable Scaling (PS) is a framework that predicts full-scale recommendation model performance from scaled-down proxies. It builds proxies with composable scaling functions along separate resource axes, then projects metrics to target scale by curve fitting with calibrated uncertainty. Of three predictors, even the raw proxy falls within repeated-run noise, and a Gaussian Process gives 15% lower relative MAE. On a task-specific ranker and a foundation ranker, PS ranks hypotheses correctly at 79-88% compute reduction.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026🧠 AI

DK-VLM adds Dual-Kernel Attention to a vision-language model. A semantic kernel keeps the pretrained attention and positional encoding. A separate geometric kernel uses estimated 3D points and camera poses to modulate attention through point displacements and relative view transformations, and a learned gate fuses the two outputs. Geometry therefore enters attention as position, not as fused 3D reconstruction features. The paper also adds Reference-Frame Composition (RC) supervision, which gives stepwise turn and heading targets derived from verified single-step annotations, to train multi-step spatial reasoning as the observer moves. DK-VLM-8B reports the highest average score across six spatial benchmarks among the compared methods, with 74.33 on VSI-Bench and 58.28 on ReVSI-32, and it preserves general video understanding.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026🧠 AI

Velox is a training-free attention router for chunked video-diffusion world models. It uses camera frustum slabs with depth intervals from streaming geometry to pick which historical token blocks to attend to, instead of letting the KV cache grow with rollout length. Frustum overlap softly modulates each layer's model-derived block importance, and uncovered regions keep the model's own routing. The fixed block granularity suits FlashAttention-style tiled execution, and slab construction runs asynchronously with denoising on the same H200. At chunk 48 it speeds up attention 4.36× and denoising 2.43×, and on the 158-case WBench navigation split it stays within 0.15 average points of dense LingBot-Fast-Cam.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026🧠 AI

GhostFaceMPC is presented as the first end-to-end face recognition system that runs both the deep CNN feature extractor and the matching step under secure multi-party computation. Raw images and biometric embeddings are never exposed at any stage, from feature extraction through storage to matching. The authors contribute an MPC-friendly quantization of the GhostFaceNet model and circuit optimizations that make this practical, and they release code on GitHub. It is relevant as an example of making CNN inference feasible under cryptographic constraints.

eprint.iacr.orgpaperPublished and added 15 Oct 2026🧠 AI

GhostFaceMPC is presented as the first end-to-end face recognition system that runs a deep CNN feature extractor (GhostFaceNet) and the matching step under secure multi-party computation. Raw images and biometric embeddings are never exposed during extraction, storage or matching. The authors contribute an MPC-friendly quantization of GhostFaceNet and circuit optimizations that make the approach practical, and they release code at crypto-unict/GhostFaceMPC.

eprint.iacr.orgpaperPublished and added 15 Oct 2026🛡️ Security

The paper fine-tunes a pretrained CNN wind-field super-resolution model with differentiable topology-inspired auxiliary losses on speed, gradient magnitude, soft high-speed level sets and local maxima. It also uses fixed-pair supervision at ground-truth persistence-pair endpoints located offline with the Topology ToolKit. A 16-configuration loss study on a fixed 168-field benchmark finds that gradient supervision gives the strongest persistence-diagram improvement, while fixed-pair supervision gives the most consistent joint improvement in persistence-diagram and merge-tree distances. The combined objective cuts bottleneck, W2,∞ and W2,2 distances by 31%, 25% and 27% relative to the pretrained CNN, at a cost of about half of standard fine-tuning's PSNR gain. It also shows that pointwise metrics and topological distances rank CNN and GAN models differently.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026🧠 AI

Presents a modular, fully onboard UAV navigation pipeline that closes the "human-aware safety gap" of geometry-only planners. A shared BLIP encoder drives captioning and text-conditioned segmentation, which is fused with depth for 3D obstacle localization. Qwen-2.5-0.5B, stabilized with RAG and grammar-constrained JSON decoding, infers per-category risk once per class and caches the verdict. Perception runs at 10 Hz on a Jetson Orin NX, and fallback LLM queries take about 0.8 s off the control path. In Gazebo it gives roughly 3x fewer collisions than a geometric planner and raises semantics-aware landing success from 34% to 96%. The 224-pixel encoder input limits small-object perception at range.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026🧠 AI

The paper shows that symmetric aggregation of a primary reward (correctness) and auxiliary rewards (brevity, efficiency) in LLM post-training can raise aggregate reward while primary-task performance falls. It proves that deterministic conditional-mean GRPO converges to auxiliary-favoring behavior when auxiliary rewards are more reliably obtained. It proposes Trust-Anchored Gating (TAG), which elects an anchor channel by policy drift and gates auxiliary credit on anchor satisfaction. On Qwen3-4B with a brevity reward, TAG reaches 0.581 pass@4 versus 0.220 for additive aggregation, and shortens responses by 44% relative to correctness-only training. Experiments on Qwen and Llama models cover math and program synthesis.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026🧠 AI