Recursive Reward Improvement via Adversarial Meta-Prompt Evolution

What it claims

Rubrics that a model writes for its own training tasks are almost always gameable. The authors classify nearly 99% of model-generated question–rubric pairs as hackable. A common culprit is a criterion phrased as "mentions that…", which an answer can satisfy by naming a keyword without the clinical substance.

RRI attacks each rubric before the solver ever trains on it. It repairs only with criteria that are shown to favour the honest answer.

Two results the summary does not carry:

• Adversary ablation. Adding the adversary to a loop that already co-evolves tasks and rewards adds 2.7 points at 9B.
• Transfer baseline. On PRBench Hard and ProfBench, Generation Only trained on the same task description gains only 1.2 and 4.7 points, against RRI-27B's 6.6 and 8.0.

How it works

RRI has five roles. A proposer, an adversary, a judge and a meta-optimizer are all frozen. Only the solver is trained.

• Self-improvement setting: a frozen copy of the solver's base model fills every other role.
• Teacher-student setting: a teacher (a GPT model in the experiments) fills them.

flowchart TD
  P[Proposer: task + signed rubric] --> H[Adversary hack answer]
  P --> O[Honest answer, rubric unseen]
  H --> J{Honest beats hack by δ?}
  O --> J
  J -- yes --> T[Training pool]
  J -- no --> R[Patch: keep only separating criteria]
  R -- "fresh probe, ≤2 rounds" --> J
  R -- "after 2 rounds, admit anyway" --> T
  T --> S[Solver GRPO]
  S --> M[Meta-optimizer]
  M -- rewrites generation guidance --> P

Proposer. It receives only a description of the target task, with no training data. It writes clinician requests, uses them as retrieval queries against a medical reference database, and then writes the task and the rubric in one call. Each criterion carries a weight in [−10, 10] excluding 0.

Hack and patch.

• The task is accepted if the honest answer outscores the exploit by at least δ = 0.20.
• Otherwise the repair step proposes criteria and the judge tests each against both answers. A negative criterion is kept only if the exploit meets it and the honest answer does not. A positive criterion is kept only if the honest answer meets it and the exploit does not.
• About four fifths of accepted criteria are positive.
• A rewrite is adopted only if a fresh probe shows a smaller gap than before.

Solver. It is trained with GRPO on rubric scores, using each benchmark's own scoring rule. It can make up to two tool calls per rollout, to the clinical database or to web search. Retrieved results are masked tool turns.

Meta-optimizer.

• After each step, an analyst labels sampled graded cases as "model failed a fair task", "generator at fault" or "model succeeded".
• It compares corpus statistics (criteria per task, negative share, use-case mix, self-similarity) against the benchmark to flag drift.
• The meta-optimizer then rewrites the guidance blocks appended to the proposer's two generation prompts. Its inputs are those summaries plus the confirmed exploits and accepted patches.

Evidence

The evaluation set is HealthBench Professional: 525 physician-written tasks covering consultation, research and writing. All scores below are with retrieval.

On raw score, the 27B Generation Only model falls slightly below the untrained model.

Overall adjusted scores for the 9B baselines:

The gains are largest on the difficult subset and on writing tasks.

Exploit taxonomy. The authors classify 12,147 exploits found across medical, legal, financial and scientific tasks:

• About one third carry wrong content that no criterion penalizes.
• About another third depend on how a loosely worded criterion is read.

How to use it

Code, prompts and configurations are released at https://github.com/DDVD233/RRI. The fetched text does not name the base model behind the 9B and 27B sizes, and it gives no licence.

Limits

• Preprint under review. The fetched text cuts off at the experimental setup, so the hyperparameters, the judge model and the appendices are not covered here.
• Unbeaten rubrics still train the solver. Tasks the adversary still beats after two repair rounds are admitted anyway. Two rounds was chosen as a compute trade-off, and δ = 0.20 is fixed with no reported sweep.
• Medical training only. All training was on the HealthBench Professional description. Out-of-domain evidence is two benchmarks at 27B.
• Retrieval dependency. Reported results use retrieval or tool calls, so performance without them is not shown here.