Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL

What it claims

FORK replaces the standard rubric-coverage reward in rubric-based RL with an F-beta score that combines rubric recall with a unit-level precision term. Under a coverage-only reward, training Qwen2.5-7B-Instruct on RubricHub-Medical shows recall saturating quickly while precision and win rate against the base model keep falling and responses grow longer. FORK reaches similar recall with rising precision. Responses are 2.6 times shorter than with the standard coverage reward. The authors also report that FORK improves satisfaction of both presence-based and absence-based rubrics. Coverage rewards from prior work hurt absence-based rubrics.

How it works

1. Unitization. An LLM splits the response into roughly sentence-sized units. It makes minimal edits, such as replacing a pronoun with its antecedent, so each unit stands alone. Greetings, headers, discourse markers and transition-only sentences are dropped. The same model does unitization and grading.
2. Credit-based necessity. For every rubric and unit pair, the grader judges whether the unit addresses that rubric. Each rubric has one unit of credit, split equally across its K matched units, so redundant units get 1/K each. A unit can collect credit from several rubrics. The grader then reads all units and their total credits and picks a threshold τ. A unit is necessary if its credit is at least τ.
3. Factual correctness. Independently, the grader labels each unit correct only if every factual claim in it is correct.
4. Precision. Precision is the fraction of units that are both necessary and correct.
5. Reward. Recall is the standard importance-weighted rubric coverage. The reward is F-beta = (1+β²)PR / (β²P + R), so higher coverage cannot offset very low precision. β > 1 weights recall more heavily. The extracted text does not state which β value was used.

flowchart LR
  R[Response] --> U[Unitize]
  U --> N[Necessity credits + threshold τ]
  U --> C[Factual correctness]
  N --> P[Precision]
  C --> P
  R --> Rec[Rubric recall]
  P --> F[F-beta reward]
  Rec --> F
  F --> Pol[Policy update]
  Pol -->|next rollout| R

FORK needs no online rubric updates. OnlineRubrics and evolving-rubric methods do require them.

Evidence

• Base policy: Qwen2.5-7B-Instruct, trained separately on two RubricHub subsets.
• Medical: 12,519 train and 1,391 test examples.
• Science: 19,806 train and 2,201 test examples, with items sourced from ResearchQA removed.
• Evaluation: in-domain test splits, plus HealthBench and ResearchQA as external benchmarks.
• Win rate against the base policy, coverage reward vs. FORK:
• RubricHub-Medical: 10.6% to 68.0%.
• HealthBench: 22.1% to 74.1%.
• Baselines:
• OnlineRubrics.
• Universal Rubrics, which adds generic necessity and correctness criteria to the predefined rubrics.
• Reweighted Absence-Based Rubrics.
• Further baseline categories are cut off in the fetched text.
• Robustness: the authors report that FORK is Pareto-optimal on coverage versus response quality. They report that results hold under different judge prompts and evaluation graders. An ablation also splits units by line breaks instead of sentences (Section 4.3).

How to use it

Source code is at https://github.com/JHKim631/FORK. Training requires an LLM grader for several jobs:

• unitizing each response,
• making a match call for every rubric and unit pair,
• choosing the threshold,
• judging correctness of every unit.

Grading cost therefore grows with the number of rubrics times the number of units per rollout. The page gives no cost or compute figures.

Limits

• Necessity judging is the weak point. The authors call it the bottleneck: every candidate training grader showed only moderate correlation with stronger reference graders on necessity, while factual-correctness judgments correlated more strongly (Table 7).
• One main base model. The main experiments use only Qwen2.5-7B-Instruct.
• Narrow domains. Training covers only the medical and science subsets of RubricHub.
• Synthetic rubrics. RubricHub rubrics are LLM-synthesized and about 90.2% presence-based.
• Truncated source. The fetched text ends during the baseline description, so full result tables, ablation numbers and the chosen β are not visible here.