Beyond the Verdict: Agentic Code Judging with Verifiable Feedback

What it claims

ARBITER is an RL-trained agentic verifier on the Qwen3.6-35B-A3B backbone. It takes a problem statement and a candidate program and returns three things: a verdict, a diagnosis, and a Python generator for a counterexample input. On the same backbone, counterexample success rises from 16.10% with no tools, to 40.61% with the agentic scaffold, to 62.67% after training. The central design claim is that a verdict-only reward collapses. Rewarding executed, validated counterexamples is what makes the verifier actually test code.

How it works

Interface. The verifier runs Python and Bash in a generic sandbox. It has no access to official tests, the reference solution, the input validator or the output checker. It ends the episode with submit(v, d, g):

• v is one of Accepted, WA, TLE, RE, MLE or CE.
• d is a natural-language diagnosis.
• g is a deterministic Python 3 program that prints one input. It is left empty for Accepted and Compile Error.

The judge then runs g, validates the input, and compares the candidate against the reference.

Training data. 20,000 candidates from online-judge-graded solver rollouts across 6,145 problems, with at most 5 rollouts per problem. 15% of candidates are accepted. 46.8% of the failures score 76–99, so they contain subtle bugs. Checkpoints are selected on a 400-candidate validation set drawn from 100 hard problems. Optimisation is GRPO in the slime framework.

Why the reward is gated. With a verdict-only reward, the policy learned to predict the majority "unaccepted" label. Mean tool calls fell from 4.9 to below one, and validation accuracy stayed at 56–58%. The fix makes most of the reward depend on executed evidence:

flowchart TD
  S[submit v,d,g] --> R{v vs reference}
  R -->|both Accepted| A[+1.00]
  R -->|disagree| Z[0]
  R -->|both unaccepted| U[+0.05]
  U --> G{g runs?} -->|yes +0.05| L{input legal?}
  L -->|yes +0.10| B{breaks C?}
  B -->|yes +0.75| T[type match +0.05]
  B -->|no| X[stop at 0.20]

The reward has further adjustments:

• A diagnosis over 1,000 tokens costs −0.05.
• A valid counterexample that overturns an Accepted reference verdict earns +0.10 extra.
• Truncated episodes are kept with zero reward. Episodes that hit the wall-clock limit are dropped from the update.

Length control. A rollout loses 0.6 reward only when all three conditions hold:

• its raw reward is at least 0.9;
• its length exceeds the upper quartile of the top-reward rollouts in its group;
• at least ¼ of the group ties for the top reward.

The penalty is switched on in alternating phases. When it was applied continuously, median length fell from 17.2k to 12.9k tokens, median turns fell from 10 to 7, and hack success dropped from 62.3% to 59.1%.

Evidence

• USACO-JUDGE (new). 900 C++17 candidates on 39 held-out USACO 2026 problems. The easy tier is 270 candidates over 18 Bronze/Silver problems. The hard tier is 630 candidates over 21 Gold/Platinum/Open problems. Candidates come from Claude Opus 4.8, GPT-5.5, GPT-OSS-120B, DeepSeek-V4, Qwen3.6-35B-A3B, plus human competitors. Mean binary verdict accuracy is 71.41% with the scaffold and 87.41% after training.
• UOJ-Bench hacking splits. 479 Easy and 1,046 Hard submissions, all faulty, re-hosted on the authors' own UOJ instance.
• False-positive penalty ablation (USACO-JUDGE hard). Setting λ=0.5 for wrongly predicting Accepted raises Accepted precision from 77.78% to 80.98%. Recall drops from 76.67% to 62.86%. The final model uses λ=0, because false negatives discard correct programs during selection.
• IOI 2026 case study. The setting is GPT-OSS-120B with a 50-submission cap per problem. ARBITER-guided refinement reaches 385.46 expected points and crosses the gold threshold in round 4. Refinement with online-judge feedback reaches 334.60.

How to use it

Code and artifacts are at https://github.com/horizon-llm/IOI2026, and the project page is https://horizon-llm.github.io/IOI2026/. The fetched text gives no install commands, weight release details or licence.

Limits

• Failing to find a counterexample does not prove a program correct. The policy can also predict Accepted without any evidence.
• The diagnosis text is never scored for correctness. Counterexample success is only a proxy for diagnosis quality.
• UOJ-Bench contains only faulty programs, so it cannot measure acceptance judgments.
• The IOI result is an expected score from a single-solver case study.
• The fetched text cuts off partway through the benchmark section. Full selection and refinement tables, and the "approaching frontier" comparison, were not visible here.