54 results for “Harness”

TraceTrial-Bench is a benchmark that scores agent task performance and agent harness reliability separately. It has 95 mechanism-oriented tasks across seven components: context, execution, governance, lifecycle, observability, tools and verification. Across 4 harnesses, 9 models and 3,420 runs, macro-averaged harness scores range from 59.8% to 72.0%, and 31.1% of runs succeed despite failing at least one harness criterion.

huggingface.copaperAdded 27 Oct 2026🧠 AI

Continual Learning Exam (CLE) is a benchmark of 120 human-authored tasks, each a five-level ladder (600 levels) across 89 professional roles and 12 occupational domains, built by 50+ domain experts. Agents must pass a level to unlock the next, and an evaluation agent scores submissions against human rubrics. Across 17 harness-model configurations, feedback plus one revision adds a median 5.3 points; the best system clears 47.7% of levels and finishes all five levels in 23.3% of tasks.

arxiv.orgpaperAdded 27 Oct 2026, published 25 Oct 2026🧠 AI

When Agents Fail: Model or Harness--Who's to Blame? proposes five measures for evaluating agent harnesses together with their underlying models: Valid Completion Rate, Model-Harness Fit, Cross-Model Fit, Interaction Residuals and Token Load. It tests four harnesses (OpenCode, Codex CLI, Pi CLI, DeepAgents) with four models on six benchmarks, across 111 settings and 52,168 trajectories. Harness rankings change by model and benchmark. OpenCode ranks first in cross-model performance.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026🧠 AI

4DMemory is a benchmark suite for long-horizon spatiotemporal memory: the LongVideo dataset (about 900 hours of video, over 5,000 manually reviewed questions) and 4DMemoryBench. It also presents the 4DMemory Harness, combining four skills, 22+ tool models and a model-specific RAG system. Across 13 models, the harness improved every one by over 50%, and the same models scored over 10% lower than on a previous benchmark.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026🧠 AI

This paper presents an offline evaluation harness for a production merchant-facing conversational agent. It stays reproducible when live backend data changes and can be regenerated when it goes stale. Used as the fitness function for GEPA prompt optimization under a correctness gate against reward hacking, it shrinks the full prompt surface by 3.5-7.6% without losing held-out correctness, across three subject-model families.

arxiv.orgpaperAdded 26 Oct 2026, published 22 Oct 2026🧠 AI

CellTrack-Bench is a benchmark that tests vision-language models on microscopic cell tracking and compares them with specialised trackers. It has 1,537 problems covering 9,041 cell-linking decisions, drawn from 15 cell-tracking datasets across four imaging modalities, with a hard subset built from failures shared by distinct algorithmic trackers. The authors compare environment harnesses, benchmark 9 closed-source and 13 open-weight models, and analyse failure modes.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026🧠 AI

Matrix-Game 4.0 is a system that separates game logic from visual rendering for playable generated worlds. A lightweight engine holds world state and rules, a multimodal coding agent writes game code and scene captions, and a distilled autoregressive video model turns low-poly proxies into detailed frames. It runs 480p at 60 FPS on one RTX 5090 with 112 ms key-press-to-display latency. Code, weights and assets are released.

huggingface.copaperAdded 22 Oct 2026🧠 AI

Paper (arXiv, 21 October 2026) on structural false negatives in LLM vulnerability discovery harnesses. Starting from Knostic's OpenAnt and raptor, then 10 more open-source harnesses, the authors map 13 points where code selection, filtering, and finding storage or merging drop vulnerabilities. One file was dropped by six harnesses before the model saw it, five without notice. It covers attacker evasion, content-steered verdicts, prompt injection, and design principles and tests.

arxiv.orgpaperAdded 24 Oct 2026, published 21 Oct 2026🛡️ Security

AgenticRoboRL is a framework for post-training robot planners across different harnesses, combining supervised fine-tuning with multi-turn reinforcement learning. Harness-specific adapters link recorded planner interactions to task outcomes, so one RL pipeline covers both VLA skill invocation and control-code generation. On LIBERO-PRO, a 9B planner rose from 18.1% to 71.0% without external memory, near Codex (GPT-5.5) with memory at 72.1%. Code and checkpoints are promised.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026🧠 AI

FunCAD-Harness is a framework in which LLM agents design CAD mechanisms and revise them using physical simulation feedback. It turns abstract functional requirements into candidate-specific physical tests. The paper also introduces FunCADBench, a benchmark of 46 mechanism families with interdependent, multi-step operations. Experiments on frontier LLM agents show it exposes mechanical failures that visual inspection misses and raises functional success over standard simulation-augmented baselines.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026🧠 AI

OpenVLHarness is a multimodal reasoning harness for MLLMs built on a capability layer (task-level tool interfaces plus evidence rendering) and persistent multimodal memory that stores images, structured data and web references as addressable artifacts. A coding interface runs generated programs over stored data. Across 22 benchmarks and eight backbones it adds 9.7-12.6 points over no-tool baselines, and GPT-6 Luna at medium effort beats Codex and Visual Sketchpad at maximum effort using at most 37% of their tokens.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026🧠 AI

VibeTraining-Bench is a benchmark for cross-component co-design, where research agents iteratively refine training data, trainers and inference harnesses for a fixed starting agent system. Tasks cover terminal operation, software engineering, multi-turn tool use and math reasoning, with safeguards against reward hacking. On SWE-bench Verified, GPT-6 Astra raises the target system from 32.9% to 53.7%, against 55.4% for the human-developed instruct model.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026🧠 AI

Self-evolving Agent Harness is a 66-page survey and theory paper on revising an agent's scaffolding (prompts, memory, workflows, tools, code, plugins) while model weights stay fixed. It frames search as evolutionary optimization and uses continual PAC-learning to derive a generalization bound, a retention bound and a high-probability admission rule with fresh validation data. It separates selection, activation, attribution and improvement as evaluation claims, and covers applications and governance. Code is at github.com/zgcagi/Harness-RSI.

huggingface.copaperAdded 19 Oct 2026🧠 AI

ToFu is a tool-aware fuzzing framework for LLM tools, including MCP servers. It maps each tool argument, using its metadata, to one of 74 CWE-guided semantics-aware sanitizers (path, URL, SQL and similar). It then generates schema-valid Atheris harnesses and semantic oracles, using runtime tracers for in-process tools and outcome oracles for opaque ones. The LLM (gpt-4.1) only maps sanitizers and writes seeds; it is never the security judge. The paper also shows a "Repeat-After-Me" trick, where an LLM told to repeat attacker-crafted text emits a tool call the orchestrator executes, reproduced on LangGraph, MAF, ADK 2.0, AutoGen, Strands and CrewAI. Across 273 runnable MCP servers (3,405 tools) it reports 253 verified bugs in 343 tools across 118 servers, such as path traversal, log injection and exfiltration. These are failures a crash-based fuzzer would miss. The result matters because it shows LLM-generated tool arguments cannot be treated as trusted.

arxiv.orgpaperAdded 18 Oct 2026🧠 AI

ToFu is a tool-aware fuzzing framework that derives semantic security checks for LLM tools (mainly MCP servers) from their metadata and a corpus of 74 CWE-guided sanitizers. It generates schema-valid Atheris harnesses, uses runtime tracers and outcome oracles to catch bugs that don't crash, and uses an LLM only to map sanitizers and seed inputs, never as the judge. It also describes a "Repeat-After-Me" attack, in which the LLM is told to repeat attacker-written text that the orchestrator then dispatches as a tool call. It was reproduced on LangGraph, MAF, ADK, AutoGen, Strands and CrewAI. On 273 MCP servers it reports 253 verified bugs across 343 tools in 118 repositories, including path traversal, log injection, exfiltration and command bypass.

arxiv.orgpaperAdded 18 Oct 2026🛡️ Security

OpenGCUA, from Alibaba's MAI-UI team, is an open stack for generalist computer-use agents (GCUAs), where the model chooses how to observe, act and organize work across GUI, accessibility-tree, API and shell interfaces. Its design combines code execution (GUI actions written as programs with loops and branches) with harness-to-model decision transfer. On OSWorld-v2 it raises mean partial score by +19.6 points across eight models, including GPT-6 Astra and Claude Opus 5, and cuts reported steps per task by 53%. The release includes the harness, prompts, evaluation trajectories, a macOS real-device harness, a cross-device SDK and an interactive lab. It also shows large efficiency gaps between models: Qwen3.8-Max used 19.4x the reasoning tokens of GPT-6 Astra on real macOS tasks.

huggingface.copaperAdded 17 Oct 2026🧠 AI

Doc2Harness is a two-stage benchmark for harness engineering. In Stage 1 a build model, working inside a fixed builder harness, gets an interface-only contract skeleton plus a development document consolidating behavior from twelve frontier harnesses, and builds a full candidate harness in an isolated container. In Stage 2 the frozen harness runs in separate per-test-case containers, driven by a scripted mock model, and is scored on four channels: requests sent to the model, files on disk, exit code and command output. Across 19 frontier models with three builds each on 328 test cases, GPT-6 Astra leads at 53.05% against 27.44% for the weakest, and failures cluster around how cross-cutting functionality is split across modules.

huggingface.copaperAdded 16 Oct 2026🧠 AI

Presents MLExpert, a 35B Mixture-of-Experts agentic model trained for autonomous long-horizon machine learning engineering. Training uses a Kaggle-MLE Factory to build executable tasks and an interaction harness, hierarchical agentic SFT with outcome-guided supervision, and Windowed Interaction RL (WIRL). WIRL treats the multi-turn decisions between successive experiment results as one execution window with a shared outcome reward, and uses asynchronous collection and policy correction. On MLE-Bench Lite the average medal rate rises from 37.9% to 72.7%, comparable to larger frontier models such as Kimi-K3 and GLM-5.3. It also improves on NatureBench Lite, agent scaffold optimization and algorithm optimization.

arxiv.orgpaperAdded 17 Oct 2026, published 15 Oct 2026🧠 AI

Harness^2 is a plug-and-play framework that recursively improves LLM agent execution harnesses in open-world settings where no ground-truth verifiers exist. Task-level recursion uses a contrastive Propose-Probe-Compose step to guide harness edits for the current task, and it scales through parallel or sequential recursion. Harnessing-level recursion distills experience from those edits into a continual-improvement harness for future tasks. A single step improves GLM-5 and Gemini 3.5 Flash over their base harnesses by up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench and 11.8 percentage points on LAB.

huggingface.copaperAdded 14 Oct 2026🧠 AI

ChainGraph is a pattern that moves exploit-chain reasoning out of the flat context window and into a Neo4j property graph. Findings, attack steps, credentials, assets and tamper-evident evidence receipts are typed nodes with pre/post-condition edges. The graph is wired to Claude Code or Codex through MCP with no custom harness. Five operator questions (next-step inference, entry-to-crown-jewel pathing, combined-severity promotion, evidence provenance, chokepoint identification) become reusable Cypher and Graph Data Science queries, plus a chain-severity score. On a synthetic multi-forest Active Directory lab modeled on GOAD, three findings rated Medium in isolation become Critical once their path to the forest root is materialized. Scoping each decision to a graph result cut the reasoning payload by about 98% on a token proxy. The authors say the results are illustrative single-run measurements, not a benchmark.

arxiv.orgpaperAdded 17 Oct 2026, published 14 Oct 2026🧠 AI

AgentSWE is a benchmark of 25 tasks covering the agent development lifecycle: Creation of an agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Builders iterate on a few illustrative cases, then the frozen submission is graded on held-out cases with requirement-derived rubrics and hard pass/fail gates. Across coding agents, models and harnesses, none comes close to delivering the requirement. Recurring failures include overfitting to visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion.

huggingface.copaperAdded 13 Oct 2026🧠 AI

Technical report for Xiaomi's MiMo-V2.6 omni-modal model family, which scales RL compute along three axes: asynchronous training with 1,568 samples and 2.7-3.7B tokens per step at up to 1M context, more diverse environments (code, general, visual, cyber) under mixed agent harnesses, and groupwise agentic grading that gives better reward signals and favors shorter, token-efficient solutions. Stability measures include a frozen MoE router and a multi-layer defense against reward hacking, built on a mid-trained hybrid-SWA architecture. The authors open-source training dynamics, RL environments and the RL framework, which makes it a useful reference for reproducing scaled agentic RL.

huggingface.copaperAdded 10 Oct 2026🧠 AI

The paper proposes evaluating LLM agents on epistemic humility, meaning their willingness to recognize, act on and communicate uncertainty. It operationalizes this as three trajectory-level dimensions (Identify, Solve, Escalate, or ISE). It tests controlled and naturally occurring knowledge conflicts, each with matched no-conflict controls, on four agents. Higher accuracy does not imply more humility: some high-accuracy agents detect conflicts but do not communicate uncertainty in wrong final answers, and agents often detect conflicts early but lose track of them later. Model-level interventions can improve humility but often cost accuracy, which suggests humility depends on the interaction of backbone model, agent harness and environment.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026🧠 AI

Introduces AgenticBBO-Bench, a benchmark for LLM-agent black-box optimization across synthetic functions, hyperparameter optimization, database tuning, chip design and molecular design under a unified finite-budget protocol. Agentic BBO beats direct LLM methods in all five domains and the best numerical optimizers in four. Ablations show extra numerical tools don't consistently help, task semantics help more than specific priors, and numerical optimizers can absorb gains from agent-found trajectories. A five-task frontier challenge evaluates seven LLMs under the Codex agent harness, with GPT-6 Astra and DeepSeek-V4.1-Flash on the performance-cost Pareto frontier.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026🧠 AI

Technical report for the MiMo-V2.6 omni-modal model series, which scales RL compute after mid-training on a multimodal corpus, using a hybrid sliding-window-attention architecture. RL is scaled along three axes: asynchronous training with 1,568 samples and 2.7-3.7B tokens per step at up to 1M context; diverse environments (code, general, visual, cyber) under mixed agent harnesses; and groupwise agentic grading that gives better rewards for long-horizon tasks and favors shorter, token-efficient solutions. Stability measures include freezing the MoE router and a multi-layer defense against reward hacking. The authors open-source the training dynamics, RL environments and RL framework.

arxiv.orgpaperAdded 9 Oct 2026, published 8 Oct 2026🧠 AI

This paper studies where tool-using LLM agents break down in the chain from gathering evidence to taking consequential actions, finding that strong static judgment of whether to act often coexists with much weaker interactive execution, and that failures arise from incomplete investigation, acting before evidence is established, or unresolved prerequisites in multi-action workflows. The authors introduce SafeActBench, a 656-case benchmark across six operational domains and five protocols, paired with a provenance-bound Evidence Ledger and a deterministic trajectory evaluator tested across ten model-harness configurations.

arxiv.orgpaperAdded 7 Oct 2026🧠 AI

This paper proposes three evaluation screens — Decisive-Action BPB, Patch MCQ, and prefix-conditioned pass@K — that predict how well a base model will perform as a coding agent after post-training, without requiring the base checkpoint to run a full agentic harness. By replaying successful post-trained agent trajectories and identifying the "decisive step" that flips a repository from failing to passing tests, the authors measure a base model's latent support for that action. Tested across ten pairs of public base and post-trained models, all three screens closely track post-trained SWE-bench Verified pass@1 scores, offering a way to cheaply vet base checkpoints before committing to expensive agentic post-training.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026🧠 AI

This arXiv paper proposes foundations for proactive LLM agents organized around three joint principles (3T: Task Capability, Temporal Allocation, Trust) and a five-dimensional design space (task scope, anticipation horizon, activation trigger, processing timing, intervention depth). It introduces PROACTIVITY-GYM, a simulation-based testbed with multi-day scenarios, stateful environments, and persona-conditioned simulated users, and reports evaluations across 23 model-harness configurations plus a 30-participant human study showing sharp trust declines after intervention misalignment and a preference for imperfect sleep-time assistance over interruptive help.

arxiv.orgpaperAdded 6 Oct 2026🧠 AI

Lachesis is a lifetime-aware KV cache placement layer for LLM agent serving that sits between the agent harness and the serving engine, deciding whether to write each KV cache segment to HBM or high-bandwidth flash (HBF) based on its expected lifetime. The authors identify temporal, structural, and inter-worker axes along which KV cache lifetime diverges in agentic workloads, and show in trace-driven simulation that lifetime-aware placement extends HBF's write-endurance-limited lifetime by 1.19–3.13x over naive HBM-first placement, reaching 3.3–12.2 device-years. This matters for serving infrastructure as agentic workloads with long-lived, multi-agent context accumulation push memory capacity demands beyond what HBM alone can economically provide.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026🧠 AI

CheckerBench is an executable benchmark of 300 tasks drawn from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems, testing whether long-horizon coding agents can synthesize working static-analysis checkers from a defect specification through repeated compile-and-analyze feedback loops. Its companion framework CheckerLab independently rebuilds submitted checkers and scores vulnerable-fixed diagnostic contrast, patch localization, false positives, and tool use; across 21 model-harness configurations mean Pass@1 is 32.30 percent and the best reaches 45.33 percent, showing reliable checker synthesis remains hard for current coding agents.

arxiv.orgpaperAdded 8 Oct 2026, published 6 Oct 2026🧠 AI