A Zeroth-Order Paradigm for LLM Preference Alignment

Hugging Face Daily Papers Papers
llm-alignment preference-optimization zeroth-order language-model optimization-method reinforcement-learning

Summary

The paper proposes Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method for LLMs that uses comparison oracles to avoid likelihood displacement. It includes theoretical guarantees and experimental improvements over existing methods.

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.
Original Article
View Cached Full Text

Cached at: 09/17/26, 06:54 AM

Paper page - A Zeroth-Order Paradigm for LLM Preference Alignment

Source: https://huggingface.co/papers/2609.19144

Abstract

Directpreferencealignmentmethodsarewidelyusedtoalignlargelanguagemodels(LLMs)withhumanpreferencesbecauseoftheircomputationalandmemoryefficiency.However,likelihooddisplacementmotivatesalternativewaystoextractinformationfrompreferencepairswithsmalllikelihoodmargins.Inthispaper,weproposeandanalyzeComparison-basedPreferenceOptimization(ComPO),azeroth-orderalignmentmethodbasedoncomparisonoracles.ComPOextractsdirectionalinformationfromthesepairswithoutdirectlyoptimizingadifferentiablepreferencelossonthem.Weestablishaconvergenceguaranteeforitsbasicofflineschemeundersmoothness,gradientsparsity,andcompatibilitybetweentheoracleandalatentobjective.WefurtherintroduceonlineComPO,whichretainstheofflinecomparisonmechanismandusesunlabeledpolicygenerationsforreverse-KLcontrolrelativetoareferencepolicy.Followingthecoverageperspectiveofpreferencefine-tuning,weestablishaperformanceguaranteeforabasicconstrainedschemeunderlocalcoverageandin-distributionpairwiserewardaccuracy.ExperimentsonMistral,Llama,Gemma-2,Qwen3,andGemma-3modelsdemonstrateimprovementsoverexistingdirectalignmentmethods,includinglength-controlledwinrates,withpair-leveldiagnosticsprovidingevidenceconsistentwithmitigatinglikelihooddisplacement.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.19144

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.19144 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.19144 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.19144 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization

arXiv cs.AI

This paper proposes a hybrid framework combining first-order safety alignment with zeroth-order refinement to enhance the robustness of LLM safety alignment against post-alignment perturbations. Theoretical and empirical results show that only a few refinement steps can improve robustness while preserving safety.

StoicLLM: Preference Optimization for Philosophical Alignment in Small Language Models

arXiv cs.CL

This research paper investigates using preference optimization (ORPO, AlphaPO) on small language models like Llama-3.2-3B and Qwen-3-4B to align them with Stoic philosophy using micro-datasets. The study finds that while 300 examples can effectively encode Stoic virtues, small models still struggle with outward-facing cosmopolitan duties.

Which Pairs to Compare for LLM Post-Training?

arXiv cs.AI

This paper studies the problem of selecting which completion pairs to label for human preference feedback in LLM post-training. It formulates comparison curation as a sampling-design problem, provides theoretical bounds on DPO's policy optimality gap, and proposes practical sampling designs that improve sample efficiency over common heuristics on synthetic and real benchmarks.