Evidence-First Agentic Verification

What it claims

Sauron, a rubric verifier from Scale AI and the University of Arizona, is built for long-horizon agent tasks. It does not accept a claim, including the agent's own "Done, report updated", until a separate component has re-read the cited source and confirmed it. Each criterion gets a MET/UNMET verdict and a receipt listing the verified evidence behind it. Code is linked at github.com/rsanayei/sauron.

How it works

1. Plan once per task. A smaller model groups the rubric criteria into clusters that are likely to share evidence. It also writes a short task "gist". The gist guides the investigation but can never be cited as evidence.
2. One investigator per cluster, run in parallel. The investigator never holds the full trace or state in context. It reaches them through bounded queries: - Broad queries find things: state_summary, state_search, trace_search, trace_tool_calls. - Targeted queries pull specifics: state_get(key), trace_ordering(e12, e35). - Per-query content caps and per-session budgets limit context use.
3. Claims are evidence atoms. Each claim is a triple of the claim text, an evidence type from a fixed set (e.g. state_value_match), and a source locator (e.g. budget.house_fund.goal).
4. The evidence verifier fetches the locator and checks the claim one of three ways: - Exact claims are compared mechanically against the stored value. - Interpretive claims go to an isolated faithfulness checker that sees only the claim and the cited passage, not the investigator's reasoning. - If the cited source cannot be retrieved, the claim is rejected and the reason goes back to the investigator.
5. Shared evidence store per cluster. It records both accepted and rejected atoms, so one trace read (e.g. a Chase account lookup) can support several criteria.
6. Verdict finalizer. It accepts the investigator's proposed verdict only if the cited verified atoms support it. When a session's budget runs out, any undecided criteria default to UNMET.

flowchart TD
  P[Cluster criteria + gist] --> I[Investigator]
  I -->|bounded queries| SRC[(State / Trace)]
  I -->|claim, type, locator| V{Evidence verifier}
  V -->|exact| M[Mechanical match]
  V -->|interpretive| F[Isolated faithfulness check]
  V -->|source not found| R[Reject + reason]
  R --> I
  M --> ES[(Evidence store)]
  F --> ES
  ES --> FIN{Finalizer: verified support?}
  I -->|proposed verdict| FIN
  FIN -->|yes| OUT[Verdict + receipt]
  FIN -->|no| I

The paper's worked example is a 51-event, 19-tool-call episode with a 30-criterion rubric. Seven bounded reads settled three criteria:

• Outcome (state): the stored house-fund goal is $20,000.
• Consistency (state and trace): the saved Amazon total of $1,147 does not match the $1,832.48 retrieved in the trace, so the verdict is UNMET.
• Process (trace): the Chase lookup happened before the total was reported.

Evidence

• Baselines: Gandalf, Agent-as-a-Judge, a snapshot-based verifier, and one more that falls in the truncated part of the page.
• PA100 (100 personal-assistant episodes, 2,586 human-labeled criteria): Sauron has the best F1 on all eight backbones. With GPT-5.6 Sol it scores 0.96 F1 against Gandalf's 0.92, at 23% lower cost. Its lead is larger on process criteria.
• PAFault (paired fault injection, 13 families, some simulating reward hacking): 97% of injected failures detected, with 2% of clean controls flagged. Baselines that were given the full trace still score lower, which the authors read as evidence that having access to the trace is not enough.
• TRAIL (detecting and localizing real agent errors): Sauron beats Gandalf and TRAIL's own judge on all three quality metrics. The fetched text does not give the values.

Limits

• Truncated page. The fetched text stops at the experimental setup. Per-backbone tables, ablations (including the effect of joint investigation), and the manual analysis of disagreements with human labels were not available.
• Budget exhaustion yields UNMET. A tight budget can therefore turn real successes into failures. Check budget settings before using verdicts as RL rewards.
• Narrow evaluation domain. The main evaluation is personal-assistant tasks only.
• Input requirements. Sauron needs the structured final state and execution trace, in the input format defined in the paper's Section B.1.
• Not stated in the fetched text: licence, install steps, and the identity of the smaller planning model.