Cutting our inference bill by 68% by being obsessive about optimization

What it claims

Cloudflare's AI code reviewer went from $2.43 to $0.77 per review, using code quality at high effort as the reference configuration. Of the $1.67 saved, $0.65 came from code changes and $1.06 from model and effort changes, measured release by release from mid-September 2026. The biggest single code saving was the first request of each session. It hit a cached prefix only 14% to 26% of the time, even though 11k to 21k tokens of tools and agent prompt were identical across every repository.

How it works

Before the work, spend was 43% cache writes, 34% cache reads and 23% output. Each call resent the whole conversation: the code quality reviewer averaged 16.6 calls, 72k tokens written and 1.08M read per review. Tokens per review had tripled from 0.91M in April to 2.99M by September. The cache hit rate rose from 85.7% to 97% over the same period. The page gives Anthropic's cache terms: writes cost 1.25x input, reads 0.1x, entries expire after five minutes unread, and a prefix is stored only where a breakpoint ends it.

The changes, in order:

1. Split the system block (September 16). OpenCode v1 built one block holding the agent prompt plus the working directory and date, with a single breakpoint at the end. The shared part was therefore never cached on its own. A marker at the end of the agent prompt, plus a plugin hook, now splits the block so the shared half gets its own breakpoint. To keep that half byte-identical, the skills list is disabled and tool descriptions are fixed across jobs.
2. Inline the standards index. The internal standards reviewer (named "Codex") had spent its first three turns reading reference files through tool calls. That index now lives in its prompt. Calls per run fell from 15.9 to 9.6 and cost per run from $1.12 to $0.72. Together with step 1, cost per review fell from $2.47 to $2.02.
3. One prefix for three reviewers (September 21 and 23). The code quality, security and performance reviewers send an identical first request: tools, a shared rules file, then the merge request with every patch. Each reviewer's own instructions come after that. A cache entry is readable only once the response that wrote it starts streaming. So two reviewers wait up to 20 seconds for the first one, saving about nine cents per reviewer.
4. Model and effort. The coordinator moved to medium effort, which made full-tier jobs 20% faster on the same model. It then moved from Opus 5 to Opus 5.5 at medium after a randomized split showed no measurable change in outcomes. High effort on Opus 5.5 cost more with no meaningful gain in findings. Lower per-token price explains only about $0.20 of the drop.

flowchart LR
  MR[Merge request + shared rules + tools] --> CQ[Code quality: writes cache]
  CQ -->|response starts streaming| SEC[Security: reads cache]
  CQ -->|response starts streaming| PERF[Performance: reads cache]
  SEC -.->|after 20s wait, sends anyway| X[Own request]
  PERF -.->|after 20s wait, sends anyway| X

Evidence

On October 3, Sonnet reviewers were split 50/50 between Sonnet 5 and Sonnet 5.5, which have the same per-token price. After 2.5 days, Sonnet 5.5 was 62% cheaper per run for code quality, 58% for performance and 47% for security. It posted about as many critical and warning findings on full-tier reviews. At medium effort, code quality made 1.7 calls per run instead of 6.2. More than half of runs used a single call and opened no other file.

A medium versus high split on Sonnet 5.5 code quality ran from October 5. High effort cost 61% more per run, about two cents per review. It posted 58% more critical and warning findings, and medium effort produced about a quarter as many acted-on findings. Code quality therefore moved to high effort.

Since early September, cache reads per review are down 82%, cache writes 54% and output 67%. Reviews over five minutes fell from 37.9% to 0.7%. Median full-tier time fell from 5m 16s to 1m 54s. A separate infrastructure change, the move to OpenCode v2, agreed with v1 on 98% of approval decisions in shadow runs and made full-tier reviews about 13% faster.

Quality was checked two ways. Authors' follow-up pushes changed the flagged lines for 59% of code quality findings, up from 34%, against a 15% coincidence baseline. Acted-on findings per 1,000 reviews stayed flat at 132 versus 145. Blinded cross-provider model judges rated 657 findings. Important findings per review went from 0.28 to 0.30. Overstated-severity findings fell from 0.46 to 0.14 per review. Runs ending without a verdict fell from 0.79% to 0.27%.

Limits

Each step was measured as weekday hours before versus after a change. The authors say this is not a statistically sound controlled experiment. The "now" quality window is one day, and each judged group held only 35 to 66 findings. A changed line shows action, not correctness.

Performance at medium effort lost real findings: about 64% fewer important ones, or one fewer every 24 full-tier reviews. Security fell by about a quarter, which is not significant. Both are still on medium, and testing them at high effort is the next step. The prefix fix depends on stripping repository-specific content, such as the skills list and variable tool descriptions, from ahead of the breakpoint.