Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

What it claims

TraceTrial-Bench scores whether an agent harness behaved correctly, separately from whether the task succeeded. It does this with three modules: Trajectory Tracing (TT), Multi-source Evidence Acquisition (MEA) and Failure Attribution (FA). The authors' headline point is that task success hides harness faults. In their 3,420 runs, 31.1% succeeded while failing at least one harness criterion.

How it works

The 95 tasks are described as expert-level and mechanism-oriented. Each has executable or state-based ground truth, so different harnesses can be compared on semantically equivalent work while each runs in its native mode, with no common adapter forcing them into one shape.

The TT module records the agent's execution trace. MEA then gathers evidence from three places: the agent's context, the harness itself and the execution environment. That evidence goes to an LLM-based evaluator.

Evaluation has two layers. One scores agent task performance and the other scores harness reliability against criteria in the seven components. FA works per run and links a failure to its cause through an explicit "mechanism chain".

flowchart LR
  A[Agent context] --> M[MEA evidence]
  H[Harness] --> M
  E[Execution environment] --> M
  T[Trajectory trace] --> M
  M --> J[LLM-based evaluator]
  J --> P[Layer 1: task performance]
  J --> R[Layer 2: harness reliability]
  R --> F[FA: mechanism chain per run]

Evidence

The grid is 4 harnesses by 9 models by 95 tasks, for 3,420 runs. Macro-averaged harness scores range from 59.8% for the lowest harness to 72.0% for the highest. The page does not name the harnesses or the models, and gives no per-component breakdown.

How to use it

The page shows a command that loads the paper into an agent through the Hugging Face CLI:

hf papers read 2610.32285

To install the CLI:

curl -LsSf https://hf.co/cli/install.sh | bash

Hugging Face lists one dataset and one Space that cite the paper, but the page does not name them. No repository, licence or run instructions for the benchmark appear on the page.

Limits

Only the abstract and an author comment are available. Neither names the harnesses or models, gives per-component or per-model results, or says how reliable the LLM-based evaluator is. Harness-reliability judgements depend on that evaluator. Its agreement with human judgement is not reported here, so it should be checked in the full paper before the scores are relied on.