Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

What it claims

When an agent harness hides long tool output behind a section index instead of pasting it into context, the accuracy loss for GPT-5.1 and GPT-5.2 comes from what the agent reads, not what it computes. Both models produce the keyed simulator value about as often under every interface. Under retrieval, though, they skip PHREEQC's formatted report (result.out) and answer from their own SELECTED_OUTPUT files. Those files print more digits, and the models then truncate instead of rounding, landing on a neighboring distractor. A one-line instruction to answer from the report recovers 6.3 to 7.9 points. An instruction to round correctly recovers nothing.

flowchart TD
  S[Agent's PHREEQC run] --> R[result.out: pH = 2.751]
  S --> O[own sel.out: 2.75065]
  R -->|inline or told to use report| B[Answers 2.751: correct]
  O -->|TOC: report skipped| T[Truncates to 2.750: distractor]

How it works

Agents get four tools (write_file, read_file with optional line range, list_file, execute_phreeqc) and up to 24 steps. They commit an answer by writing it to a file. Only what execute_phreeqc returns changes between conditions:

• Raw100k: the report inline, head/tail-truncated to 100,000 characters.
• TOC: a section index with starting lines; the agent reads line ranges.
• Grep: metadata only, plus a regex search tool.
• Hybrid: inline when the report fits in 100k characters, TOC otherwise.
• TOC+Report and TOC+Round: TOC plus a three-sentence instruction.

The authors re-ran all 11,007 simulator calls from 5,773 logged trajectories offline. For each of the 137 numeric items they then scored three stages. Build: did the agent's own run print the key? Exposure: did that value reach the agent? Closure: given that it saw the value, did it answer correctly? A Shapley decomposition splits each accuracy change across these stages. The authors note it allocates the difference and does not estimate causal effects.

Evidence

Inline vs TOC accuracy (headline runs) is 65.8% vs 57.0% for GPT-5.2 and 67.7% vs 57.0% for GPT-5.1. Opus 4.6 scores 88.0% vs 87.3% and Sonnet 4.6 scores 91.1% vs 91.8%. After Holm correction over five models, only GPT-5.1's pooled loss is significant (p = 0.005). GPT-5.2's is not (p = 0.128). With the simulation frozen at GPT-5.2's first successful execution and only the delivery changed, TOC still costs 9.4 pp [−14.5, −4.6] over three paired repeats.

Grep does no better than TOC (57.6% GPT-5.2, 48.1% GPT-5.1), so the cost is not specific to the index format.

Truncation drives closure failures. When a seen value rounds to the key but truncates to a neighbor, GPT-5.2 picks the neighbor in 26/35 TOC trajectories and GPT-5.1 in 19/27. Sonnet does so in 1/35. Under TOC, GPT-5.2 and GPT-5.1 receive no part of the report in 60.3% and 53.0% of trajectories. Sonnet skips it in 10.5%.

GPT-5.4's net change is +0.2 pp, but that figure hides about ten points lost to evidence substitution on numeric items. More frequent correct simulations under retrieval offset the loss.

TOC+Report cuts report skipping below 2% and raises closure to 99% without changing build. Hybrid gives GPT-5.1 and GPT-5.2 the same benefit at similar or lower input-token cost (59.7k vs 65.7k per item for GPT-5.2). Simulator access adds 15.8 to 45.6 pp over no-tool baselines.

Limits

The study uses one simulator and 158 items from 19 setups, all written by one domain expert. The authors kept 158 of 200 items after re-execution and dropped 42 that the author confirmed as defective. Keys come from PHREEQC 3.8.6 with a fixed database. Distractors differ in the last printed digit, which makes this task unusually sensitive to rounding.

GPT and Gemini models ran at API defaults with no reasoning-effort setting. Claude ran at temperature 0 without extended thinking. Gemini 3 Flash Preview was excluded because it gave no parseable answer on 46% of items inline and 71% under TOC. The 33 long-report items share one TOC run because the original index was empty for them. Automatic and manual stage labels agree on 52 of 58 hand-checked trajectories. The authors call "evidence substitution" descriptive: agents may read their own files simply because those files are short.