Reproducible Evaluation as the Substrate for Safe Prompt Compression in Production Agents

What it claims

DoorDash's Mx ML team built an offline evaluation harness for a merchant-facing conversational agent in production. The harness keeps benchmark datapoints reproducible while live backend data changes. It then uses that harness as the fitness function for GEPA prompt compression. Across three model families, the full prompt surface (36,336 tokens) shrank by 3.5% to 7.6%. Only the Anthropic run showed a statistically clear correctness gain on held-out data; the other two showed no detectable change. The paper is a NeurIPS 2026 workshop paper (Agents in the Wild), posted October 2026.

How it works

Each tool gets a mutability class, and each class gets its own reproducibility mechanism:

• Settled reads, data that is immutable once settled: the harness stages the clock.
• Current-state reads: it freezes the observed payload.
• Writes: it seeds state, then reads it back. Write tools are graded on the end state they leave, not on their text.

Datapoints come from a generator agent that sees exactly the prompt and tools the production agent sees on that turn. A datapoint is kept only if its tool-call trace runs end to end and the answer follows from it. A human then reviews every surviving datapoint for well-posedness and spec fit. Generation is stratified by skill and archetype. The whole pipeline, including human review, can be re-run against current backend state when the benchmark goes stale. Deterministic judges and LLM-rubric judges score correctness and policy compliance.

For compression, GEPA treats the base system prompt, all skill instruction bodies and all tool descriptions as one joint candidate. It edits every component in each reflection step. The authors argue that compressing one component alone only relocates duplicated text. Unlike typical GEPA use, the seed is spec-complete, so accepted mutations remove text rather than add it.

flowchart TD
    C[Candidate prompt surface] --> G1{Violates mandatory rule?}
    G1 -- yes --> Z[Fitness = 0]
    G1 -- no --> G2{Any skill mean >0.02 below seed?}
    G2 -- yes --> Z
    G2 -- no --> F[Val score minus tiny token penalty]
    F --> P[Hybrid Pareto frontier]
    P --> C
    F --> S[Final pick: max fitness on val only]

Fitness is zero for any candidate that breaks a mandatory rule. The authors made this a hard gate because a soft penalty would reward deleting rarely exercised instructions, a form of reward hacking. A no-harm gate also zeroes any candidate whose per-skill mean falls more than 0.02 below the seed's. Otherwise, fitness is the validation score minus ε times token count, with ε = 10⁻⁶. This penalty only breaks exact ties.

Evidence

The data covers 10 skills: 302 train, 99 validation and 196 held-out test datapoints, generated with seed 42. Each family ran GEPA with a rollout budget of 800. The test Δ is winner minus seed, the mean of 5 rollouts per datapoint, with a 10,000-sample bootstrap 95% CI.

Limits

• Drift not measured. The benchmark is a frozen snapshot, so the authors have not measured how a compressed prompt's advantage decays as production data drifts.
• Gain depends on the model. The size of the gain varies by subject model, and the authors read this as depending on the model, not only on the harness.
• Out-of-scope calls go unflagged. A tool call routed outside a datapoint's staged scope hits live, unstaged state and scores as an ordinary wrong answer. A failed seed, by contrast, is hard-gated to 0.
• Regeneration has a human cost. Regeneration requires human review of every datapoint.
• One system only. Results come from one proprietary agent.
• No release. The page mentions no code, data or licence release.