The triage is the product: running AI agents against Ethereum's protocol code

What it is

A methods writeup from the Ethereum Foundation Protocol Security team on running coordinated AI agents against protocol code. The summary already covers the CVE, the fleet-coordination pattern, the false-positive taxonomy, and the headline practices. What it omits is the concrete triage machinery — the candidate schema, the two-question validation test, and the specific limitation where agents fail — which is the reproducible part.

The candidate schema

Before anything counts as a finding, an agent must fill a fixed schema. This is the enforcement mechanism behind "mandatory reproducers" — it forces a testable claim and a definition of done:

• target: component and entry point an attacker can actually reach
• invariant: the property that must hold
• mechanism: the specific way it might break
• success: observable proof — a panic, a stall, an accepted-invalid input
• reproducer: self-contained artifact that runs against the real code
• dedup: a key so two agents don't chase the same thing

The two-question triage

Every surviving candidate gets two independent checks:

1. Can a real attacker reach it in a normal configuration?
2. What does it cost the attacker to trigger versus what it costs the network if it works? (Any-single-peer triggers rank far above bugs needing special access or heavy resources.)

Everything is also checked against a running list of known/fixed/rejected issues, or agents re-report closed bugs endlessly.

The limitation worth knowing

A single agent session is good at one-shot reasoning and bad at sequential/stateful bugs — where each step is valid and only the ordering is wrong. For these the agent should not be the search tool; its job is to suggest which sequences to run through a stateful test harness. Used as a replacement for the harness, it misses the most expensive bugs there are. Stanislav Fort's "jagged frontier" framing applies: one good result on one codebase predicts nothing about the next.

Numbers cited (from others, not their own)

• Anthropic's property-testing agent: ~1,000 candidate reports, ranked and expert-reviewed down to a top tier that held up ~86% of the time.
• A repository-context-files study: extra requirements lowered task success and raised cost by over 20% — the basis for "norms, not scripts." They decline to publish their own acceptance rates, arguing target-specific numbers describe the target, not the method.