Tag
SAGE proposes a group-level uncertainty target that constructs an answer-conditioned uncertainty geometry over sampled responses to improve verbal uncertainty alignment in LLMs, and introduces GUPO for training. Experiments across reasoning tasks show improved uncertainty ranking and reduced overconfidence.
This paper presents a systematic mechanistic analysis of six preference optimization methods (PPO, DPO, SimPO, ORPO, GRPO, KTO) across three open-weight model families, using probing and sparse autoencoders to reveal how alignment algorithms reshape internal representations in qualitatively distinct ways.
DOG-DPO is a training-free data selection framework that treats preference pairs as structured geometric signals, decomposing multi-dataset preference geometry into anchor and residual subspaces to select diverse subsets for safety alignment. It achieves strong utility-robustness trade-offs using only 11% of preference pairs across six safety benchmarks.
MetaEvo proposes a two-stage framework for continual evolution of LLM-based agents, using preference-based optimization to enhance principle abstraction and modular architecture for experience reuse, outperforming strong baselines on reasoning benchmarks.
Z-Reward is a teacher-student framework that decouples complex reasoning from efficient reward deployment for text-to-image training. It achieves 89.6% human preference accuracy with a 27B teacher and 88.6% with a 9B student, outperforming prior methods.
A comprehensive guide to 15 policy optimization and preference optimization techniques important in 2026, including GRPO, DPO, REINFORCE++, and many newer variants, mapping the landscape of reasoning RL methods.
This paper introduces Preference Delta Aggregation (PDA) and Geometric Alignment Merging (GAM) to aggregate multiple 'weak' preference signals from weaker model pairs via LoRA merging, improving strong LLMs on knowledge reasoning and agentic search tasks by over 6% on average.
Introduces HDSR and HDSR-PL, methods that use hallucination detectors to guide iterative self-refinement and preference learning, achieving up to 48% reduction in hallucinations for clinical summarization using Llama and Gemma models on MIMIC-IV-Note.
This paper introduces BiDPO, a framework that enhances text-to-image models for complex compositional prompts through preference-based fine-tuning and region-level guidance, achieving state-of-the-art results on compositional fidelity benchmarks.
Presents WeCon, a weight-conditioned neural solver for multi-objective combinatorial optimization problems that achieves comparable hypervolume to the state-of-the-art while reducing inference time by 40%.
This paper proves that the equivalence between Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF) is conditional and often violated in practice, revealing failure modes where DPO optimizes relative advantage rather than absolute alignment. The authors introduce Constrained Preference Optimization (CPO) for provable alignment and demonstrate state-of-the-art performance.
Proposes AttentionPO, a token-weighted direct preference optimization method that uses attention from the LLM itself to estimate token weights, improving alignment performance on AlpacaEval, MT-Bench, and ArenaHard without requiring a separate reward model.
This paper proposes CTO, a method that improves code translation by combining syntax-guided and semantic-aware preference optimization through contrastive learning and direct preference optimization, achieving significant improvements over existing baselines in C++, Java, and Python translations.
This paper analyzes spurious correlation learning in preference optimization methods like DPO, identifying mechanisms such as mean spurious bias and causal-spurious leakage. It proposes 'tie training' using equal-utility preference pairs as a mitigation strategy to reduce reliance on spurious features without degrading causal learning.
This paper introduces YFPO, a neuron-guided preference optimization framework that uses internal activation signals to improve mathematical reasoning in large language models.
The paper introduces Macro, a preference alignment framework using DPO to improve the validity and minimality of self-generated counterfactual explanations across multiple languages.
This research paper investigates using preference optimization (ORPO, AlphaPO) on small language models like Llama-3.2-3B and Qwen-3-4B to align them with Stoic philosophy using micro-datasets. The study finds that while 300 examples can effectively encode Stoic virtues, small models still struggle with outward-facing cosmopolitan duties.
This paper introduces a topology-enhanced alignment framework for LLMs, utilizing trajectory topology loss and topological preference optimization based on persistent homology to regularize semantic trajectories in hidden space.
This paper introduces PNAPO, an offline preference optimization framework for rectified flow models that augments preference data with noise samples and uses dynamic regularization to improve training efficiency and sample efficiency.
FSPO proposes a few-shot preference optimization algorithm for LLM personalization that reframes reward modeling as meta-learning, enabling models to quickly infer personalized reward functions from limited user preferences. The method achieves 87% personalization performance on synthetic users and 70% on real users through careful synthetic preference dataset construction.