Tag
该论文提出 PERPL 框架,通过 LLM 模拟读者的相对偏好来学习个性化事实核查解释改写,在保持事实忠实性的同时优于更大的通用 LLM。
A tutorial-style article accepted to IEEE Signal Processing Magazine that surveys the mathematical foundations and algorithmic frameworks for aligning LLMs with human feedback, covering SFT, RLHF, DPO, and their links to inverse reinforcement learning and discrete choice models. It serves as a comprehensive resource for researchers and practitioners on alignment methodologies.
The paper proposes DeCert (Decision Certificate Supervision), a framework for training generative reward models that evaluates the validity and verdict-consistency of generated rationales without needing reference explanations, improving rationale quality and positional-bias robustness by up to 14.8 points.
This paper introduces Symbolic Preference Labeling, a neuro-symbolic method where an LLM classifies predefined prompt/response attributes and explicit rules derive auditable, reproducible preference labels, then trains a policy via DPO and reports refusal rates on benign and toxic prompts.
该论文首次研究了可复制的对抗性多臂老虎机(replicable dueling bandits)问题,设计了在共享内部随机性下保证每次运行选择一致比较对的算法,并在多种随机传递性假设下证明了次线性后悔界,实验在合成数据和语言模型偏好上验证了可复制性。
This paper shows that Luce's choice axiom, assumed by the Bradley–Terry model underlying Chatbot Arena rankings, fails for near-copy models: votes between similar models are up to 1.37× more decisive than Bradley–Terry predicts, based on 1.8 million Arena votes. A small four-to-six parameter correction improves held-out vote prediction, though leaderboard order barely changes.
The paper introduces TasteBench, an expert-gold pairwise benchmark for measuring 'taste' in VLMs across visual, brand, book, and tech domains, along with a control protocol that quantifies and corrects evaluation artifacts like position bias, lexical leakage, and difficulty ceilings.
该论文提出福利感知偏好获取方法(WAPA),在有限人工反馈预算下,通过社会 welfare 目标来分配人工标注,以改善多元对齐中不同个体偏好的监督质量,实验证明仅获取 20% 人工标签即可恢复 75–86% 的效用差距。
The paper proposes Approximate Pareto Optimality (APO), a method for collaboratively learning lightweight LLM aligner initializations that support few-shot personalization under data deficiency, grouping users by compatible updates and coordinating competing objectives toward Pareto-optimal points with proven suboptimality bounds.
该论文提出 Robust Nash Alignment,一个面向不确定成对偏好的博弈论对齐框架,通过主-对偶代理博弈与乐观镜面下降-上升算法,实现了对最坏情况胜率的认证下界,并在 LLM 对齐实验中优于基线方法。
This paper introduces a preference-guided adaptation framework for open-vocabulary semantic segmentation that leverages prompt disagreement to generate binary preferences, eliminating the need for dense pixel-level annotations and improving performance in specialized domains.
The paper introduces a multi-stage architecture for learning individuated utility functions from multi-modal data, showing that models accounting for heterogeneous preferences outperform universal utility models in subjective tasks like aesthetic judgments.
SKIP is a self-knowledge-guided step-wise preference learning framework that improves reasoning compression in large language models, mitigating performance degradation and reducing overthinking by using DPO to guide efficient reasoning.
This paper investigates the self-consistency of cardinal LLM preference judgments across multiple models and finds large persistent inconsistencies, suggesting they cannot be faithfully summarized by a single utility function.
FedPref introduces a federated preference learning method for structured radiology report extraction, allowing institutions to collaborate without sharing data while improving extraction accuracy with gains in F1 scores on simulated hospitals.
This paper recasts preference-based reward learning as a human-autonomy team problem, arguing that a teacher who knows the objective can design more efficient training examples than learner-driven query selection. It introduces understanding statements with second-order theory-of-mind to keep the teacher's model of the learner synchronized, showing in simulation that this approach outperforms learner-led selection.
This paper introduces KappaSharp, a method for condition-number shaping in Preferential Bayesian Optimization that corrects the ill-conditioned Hessian from isolated pairwise comparisons, showing up to +10.9% improvement over the standard PairedGP/EUBO baseline on 11 benchmarks including plasma medicine controller tuning.
Crayotter introduces Group-Relative Preference Backpropagation (GRPB), a method for training long-horizon video editing agents using within-task preference rankings rather than global scalar rewards. The resulting 9B model outperforms several proprietary systems on AgenticVBench.
This paper proposes a preference-based learning framework for antibody expression ranking, integrating scarce quantitative data with large-scale weak positive supervision from immunization sequences. The method adapts Direct Preference Optimization to protein language models using a union-masked log-likelihood approximation and IMGT-based alignment, achieving improved ranking performance on a diverse internal dataset.
Introduces 'weights to words', a method that automatically discovers domain-relevant preference dimensions described in natural language from choice data, enabling users to inspect and edit preference model inferences in real time.