Fighting AI Slop in Production Codebases: How Huntress Improved Fable 5.1’s API Recall

What it claims

Huntress measured whether Claude Fable 5.1 in Claude Code reaches for built-in Rails APIs instead of hand-rolling equivalents, and the headline 48%→100% arc hides the finding that matters: the 48% baseline is really two populations. Features the model already knew (generates_token_for, normalizes) hit 83% (10/12); features it never reached for (perform_all_later, comparison:, has_secure_token) hit 0% (0/9). That split, not the average, dictates which intervention works.

How it works

Three levers, each tested in every combination (round two: 8 setups, 4 tasks, 4 attempts, 128 trials):

• CLAUDE.md rule (four lines: name the Rails version, prefer framework APIs, grep the installed gem before writing a helper, justify exceptions in the PR).
• Lookup tool — bin/rails-api search/show, a script that reads the pinned bundle and prints a method's signature, options and source.
• Reviewer — a Claude Code Stop hook that sends the finished diff to Sonnet 5 with one question ("does this hand-roll something installed Rails provides?"), exits 2 to block on REWORK, so the agent continues.

The reason the reviewer succeeds where the rule fails: it works from finished code, not the agent's sense of what looks framework-native, so it catches slop the agent doesn't recognize as slop.

Evidence

Paired McNemar tests, six-discordant-pair floor:

• Rule alone: 86% (18/21, p=0.008, zero regressions).
• Head-to-head wins/losses: rule 23-1, reviewer 24-0, lookup tool only 6-2 ("only works when the rule makes the agent use it").
• Reviewer took bulk-enqueue (perform_all_later) from 0/16 to 16/16.
• All three combined: 28/28 across seven tasks, at ~30¢ more per task.

Notable negatives: the popular "Rails Way" checklist scored 71% and made the agent worse — faster, never ran RuboCop, shipped autocorrectable lint on three correct solutions. An optional skill was never invoked in 27 trials. Optional items get ignored; the rule must be mandatory context.

How to use it

Harness is a fork of rails/lemans (the Rails team's benchmark harness) with a Claude Code adapter. The full CLAUDE.md rule text, lookup-tool paragraph, and reviewer prompt are in the appendix and reproducible directly. Setup tested: Fable 5.1 at default effort, one Rails 8.1.3.1 app on Ruby 4.0.6, hidden RSpec + app specs + RuboCop as grader.

Limits

Authors flag it: small sample, few APIs, seven tasks. Recall scored by regex on the diff. Four of seven seed tasks were themselves written with Claude Code. Not yet run on Opus 5 or open-weight models — stated as a next step.