Terminal-Bench 4.0

What it claims

Terminal-Bench 4.0 is a maintenance/re-calibration release of the terminal-agent benchmark, not a "sequel" with new tasks. It calibrates per-task compute resources (time, CPU, memory), fixes 19 tasks, and removes 8 tasks. The stated effect: fewer agent timeouts and errors than 3.0, so less measurement noise. Remaining errors are "largely model refusals and output-token-exceeded errors."

What actually changed

• Resource calibration: all tasks now set to a flat 8-hour agent timeout. Method follows the paper Quantifying infrastructure noise in agentic coding evals — run with large resource headroom, then pick a setting that doesn't move the final metric. Frontier models now rarely hit timeouts.
• 8 tasks removed: 2 saturated, 2 refusals, 2 with public solutions, 2 with unresolved quality/platform issues. "Saturated" is defined precisely: every class in every family of the latest model generation solves it 5/5.
• 19 tasks fixed: instructions, environments, and verifiers updated for flakiness or misspecification.
• Versioning is now semantic and continuous, not sequential releases. This is 4.0 (not 3.1) because resource changes plus task removals are breaking changes requiring re-running trials.

Evidence you can act on

The one concrete model observation: Sonnet 5 used 21.6B tokens on its leaderboard run vs 6.5B for Opus 5, and sometimes hit timeouts and output-token-exceeded errors. The 128k max-output-tokens setting was not enabled (which might have helped), and Sonnet 5 showed large variance in execution time. This is a useful data point if you're comparing agent cost/token efficiency rather than just pass rate.

Practical implications

If you have prior TB 3.0 numbers, they are not comparable — re-run against 4.0. Built on the Harbor framework, available on Harbor Hub (no install commands given on this page). Per-task changes are in each task's README changelog and the GitHub v4.0 milestone.

Roadmap / limits

No actual leaderboard numbers or pass rates are given on this page beyond the token-usage figures — for scores you must go to the leaderboard/Harbor Hub. Upcoming: 4.1 = tamper-resistant verifiers; 5.0 = new tasks. The page flags that current verifiers are not yet tamper-resistant, and that there's still "lots of room to improve on both cost and performance."