The Agent Said It Was Done. The Database Disagreed.

What it claims

Grading agents on the backend state they leave behind exposes failures that tool-call and final-reply checks miss. Repetition then separates breadth from dependability, and the two rank models differently. The page adds what the summary omits: how failed trials break down, how much pass@1 survives 20 repeats per model, dollar figures for cost per success and per dependable task, and a runnable OpenEnv procedure with pinned releases.

How it works

• Task definition. Each task specifies:
• a starting backend state
• a user goal
• the available MCP tools
• a domain policy
• executable checks over the terminal state
• Simulated user. It holds private context, such as a booking reference or date of birth, and reveals it only when the agent asks.
• Isolation. Every attempt gets an isolated MCP session with freshly initialized state. No database rows or cached tool state are shared across the 20 trials.
• Grading. A side-effect extractor derives what changed. Deterministic judges accept any trajectory that reaches the required end state and reject wrong, missing or extra effects.
• 477 of the 507 tasks are graded on state alone.
• 30 add a narrow binary response rubric, for example "did the agent disclose this is not guaranteed?"

flowchart LR
  subgraph Model side
    A[Task, dialogue, tool schemas] --> B[Agent]
  end
  B --> C[Isolated MCP session]
  C --> D[Side-effect extractor]
  subgraph Evaluator side
    G[Golden state, assertions, credentials]
  end
  D --> J[Deterministic judges]
  G --> J
  J -->|30 tasks| R[Binary response rubric]
  J --> P[Pass / fail reward]
  R --> P

The model never sees the golden state, assertions, grading internals or credentials.

Evidence

Clean-looking failures. The common set covers 121,680 valid trials over 12 models, with 79,853 failures. Of those failures, 67.24% terminated cleanly, called a state-changing tool and reported no tool error. Among them the checks found:

• wrong field values in 77.61%
• unintended extra effects in 43.30%
• missing required effects in 25.36%

These categories overlap.

Single-attempt scores (pass@1, 18 models).

• Claude Opus 5.5 leads at 67.16%, ahead of Claude Opus 5 at 66.50%.
• Kimi-K3 is the strongest open-weights model, within a point of GPT-6-Astra.
• Domain matters as much as model: retail averages 59.52% and auto insurance 33.83%. Claude Opus 4.6 scores 68.62% on retail but 8.30% on auto insurance.

How much pass@1 survives 20 repeats.

• GPT-6-Astra keeps 78%, the best retention. The summary's "Opus most dependable" framing omits this.
• Claude Opus 5.5 and Opus 5 each keep 71%.
• GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8%.

Breadth versus consistency.

• Kimi-K3 solves 476 of 507 tasks at least once, but only 68 (13.41%) pass all 20 attempts.
• Claude Opus 5 solves 79.09% at least once, but 47.53% pass 20/20.
• Opus 5.5 and Opus 5 both pass exactly 241 tasks 20/20. Opus 5.5's higher pass@1 bought no extra dependability.
• The authors' advice is that pass@20 (best-of-k) is the wrong column for work that touches real records.

Cost. Prices are undiscounted OpenRouter list rates from a 2026-09-20 snapshot, with quantized endpoints excluded. Opus 5.5 pricing comes from Anthropic's site. The authors call these comparative indices, not invoices.

• Cost per successful attempt. Three models are on the Pareto frontier:

Claude Opus 5 ($0.475) is dominated by Opus 5.5.

• Cost per task passing 20/20:

The cheapest model per single success is not the cheapest per dependable task.

Failure signatures. About four in five failures are tool handling, not reasoning. The typical failure is not recovering from tool errors, failed preconditions or empty lookups. These figures are unweighted averages of per-model shares, not causal attributions. The detailed breakdown is in Table 5 of the paper and does not appear on this page.

How to use it

Requirements:

• Linux or WSL, Python 3.11+, uv and Docker.
• Model endpoints for the agent, the simulated user and the judge. One endpoint can serve all three roles.

Install:

# 1. OpenEnv + the ThinkingBox environment
git clone https://github.com/huggingface/OpenEnv
cd OpenEnv
uv sync --project envs/thinkingbox_env --frozen
# 2. The executable benchmark, at the pinned release
git clone https://github.com/microsoft/thinkingbox-data
git -C thinkingbox-data checkout thinkingbox-bench-v1.0
# 3. The ThinkingBox CLI, which provides `tb`
uv tool install "thinkingbox @ git+https://github.com/microsoft/thinkingbox"

Start Typesense 30.1 (second terminal):

mkdir -p .typesense-data
docker run --rm -d --name thinkingbox-typesense \
  -p 8108:8108 \
  -v "$PWD/.typesense-data:/data" \
  typesense/typesense:30.1 \
  --data-dir /data --api-key=Fake --enable-cors
until curl -fsS http://127.0.0.1:8108/health; do sleep 1; done

Start the MCP servers (third terminal):

cd OpenEnv
tb mcp-start --host 127.0.0.1 --port 7111 \
  --servers "$PWD/thinkingbox-data/servers/servers.yaml"
curl -fsS http://127.0.0.1:7111/health

Start the OpenEnv server (first terminal), check readiness, then score one task:

OPENENV_TB_CONFIG="$PWD/thinkingbox.yaml" \
uv run --project envs/thinkingbox_env --frozen server
curl -sS http://127.0.0.1:8000/ready
echo "- sandbox_external_retail_group1.py:test_case_ST002_001" > one_task.yaml
uv run --project envs/thinkingbox_env thinkingbox-eval \
  one_task.yaml \
  --config "$PWD/thinkingbox.yaml" \
  --output results.jsonl \
  --errors-output errors.jsonl \
  --repeat 1 --message-timeout 1800

The thinkingbox.yaml file names your three models; the page links a config guide for its format.

Licences: the code is MIT, the benchmark data is CDLA-Permissive-2.0, and the OpenEnv environment is BSD-3-Clause. An RL training repo is listed as "coming soon".

Limits

• Readiness check gaps. /ready returns 503 until its own checks pass, but it cannot see Typesense or live-probe every model endpoint. Verify those yourself.
• Example script is partial. example_usage.py only resets the environment and lists tools. Use thinkingbox-eval for actual scoring.
• Operational errors. These go to errors.jsonl. A canonical result must rerun them or account for them; the authors counted them as failures.
• Synthetic tasks. Every task is a synthetic reconstruction, and the customers are not real.
• Untested mitigations. The suggested fixes (classifying errors for targeted retries, a smaller tool surface, human approval for irreversible changes) were not measured on the benchmark.
• Evaluation-only adapter. The adapter is built for evaluation. Training should use separate, non-benchmark scenarios.