A Zeroth-Order Paradigm for LLM Preference Alignment

What it claims

ComPO is a zeroth-order preference-alignment method: instead of optimizing a differentiable preference loss (as DPO does), it uses comparison oracles to extract directional information from preference pairs, targeting the likelihood-displacement problem that arises when preference pairs have small likelihood margins. Beyond what the summary states, the abstract names the specific assumptions under which the guarantees hold — these are the actionable content of an abstract-only page.

How it works

Two schemes:

• Offline ComPO: the basic comparison-oracle mechanism. Its convergence guarantee rests on three stated assumptions — smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. The gradient-sparsity assumption is notable: zeroth-order methods generally scale badly with parameter dimension, and sparsity is the lever that makes the guarantee plausible for LLM-scale models.
• Online ComPO: retains the offline comparison mechanism but adds unlabeled policy generations for reverse-KL control against a reference policy. Its performance guarantee holds under local coverage and in-distribution pairwise reward accuracy — i.e., the reference/coverage conditions typical of the coverage view of preference fine-tuning, not global assumptions.

Evidence

The page is abstract-only and reports no numbers. It states qualitatively: improvements over existing direct alignment methods including length-controlled win rates, across Mistral, Llama, Gemma-2, Qwen3, and Gemma-3; and pair-level diagnostics "consistent with mitigating likelihood displacement." No baselines, magnitudes, or benchmark names (e.g. AlpacaEval) are given on this page.

Limits

• Abstract only — no quantitative results, no compute or training cost, no code or model release mentioned on the page. To judge whether the win-rate gains are material, you must open the PDF.
• The guarantees are conditional on assumptions the authors themselves list (gradient sparsity in particular); whether those hold in your setting is not addressed by the abstract.
• Online performance guarantee is for "a basic constrained scheme," suggesting the theory covers a simplified version, not necessarily the full method run in experiments.

If you are evaluating zeroth-order alternatives to DPO specifically to address likelihood displacement, this is worth the PDF read for the assumptions and diagnostics; the abstract alone does not establish the size of the gains.