Tag
This paper identifies 'common-mode hidden offsets'—shared directional shifts in hidden states during preference optimization—as the mechanism behind vocabulary well formation and generation collapse, showing it occurs across 17 model lineages and offering a representation-level early warning signal for ranking-metric blind spots.
This paper proposes SPOKE, a strategic preference optimization method with KL-anchored equilibrium, to teach compact LLM-based agents (SmallThinker-4B) to play poker more strategically under hidden information, achieving lower exploitability than SFT, DPO, GRPO, and GSPO in Kuhn and Leduc poker.
The paper introduces calibrated accommodation pairs (CAP), a preference dataset that rewards chat models for matching users' dialect while preserving Standard English, and audits how SFT, DPO, and RL stages change a model's tendency to accommodate informal dialect cues across 10 open model lineages.
提出了PsyMemBehav，一种基于心理学画像与记忆检索的两级身份到动作推理框架，用于模拟社交媒体用户行为，并构建了涵盖5,862名真实用户、557万条行为的基准数据集，实验显示其27B模型在多数个体级指标上超越GPT-5.4。
DeLoop is a self-supervised preference optimization method that reduces repetitive tool-call loops in long-horizon language-model agents by using the agent's own rollouts as training pairs, cutting looping tasks from 57.3% to 6.1% under context compression and improving success rates on larger models.
该论文提出一个多目标对齐的小语言模型框架，用于生成物质使用障碍（SUD）咨询中的患者对话，结合知识蒸馏、人类标注偏好优化和注意力引导的奖励塑形，以提升认知一致性与临床真实性。
The paper proposes Pluralistic Preference Optimization (PlurPO), a post-training method where language models simulate multiple stakeholders in interpersonal conflicts and are trained to prefer responses acceptable to all of them, substantially reducing social sycophancy across four model families without requiring ground-truth labels.
This paper formalizes personalized preference learning as a geometry-aligned optimization problem and proposes GAP-DPO, an iterative algorithm that performs utility-aware, gradient-aligned preference pair selection, showing improved stylistic fidelity and preference alignment over standard DPO variants.
本文提出一种通过组合偏好奖励（Bradley-Terry目标）与基于评分标准（rubric）的奖励来对前沿文生图模型进行后训练的方法，避免简单加权平均导致的次优优化，在Arena文生图排行榜上RL训练的Flux2dev比基线Elo高69分，后训练的Ideogram-4以1223.5的Elo超越所有开源模型，并发布Arena-T2I-Training 1K训练数据子集以支持可复现研究。
The paper introduces MetaSteer, a method that learns nonlinear, context-dependent interventions applied to attention projection matrices in LLMs, trained once on pooled preference data and transferred zero-shot to unseen concepts and agentic settings, matching or outperforming task-specific steering baselines.
BiasReducer is a lightweight framework that edits only the linear reward head of reward models to adaptively mitigate biases toward superficial attributes like response length and confidence, using a sparse-autoencoder-style encoder to detect and rank relevant biases per dataset. Across five reward models it improves three benchmarks by 8.3, 18.0, and 6.9 percentage points on average, outperforming training-based baselines and reducing downstream verbosity and sycophancy.
This paper investigates how to predict objective conflicts and cover trade-offs in steerable pluralistic alignment using Multi-Objective Direct Preference Optimization (MODPO), showing that pre-training measurements can predict alignment for human-annotated data and providing methods for broader trade-off coverage.
The paper proposes DiaWhisper-DPO, an end-to-end model for transcription and role attribution in clinical interviews using failure-mined preference optimization, achieving high accuracy and reducing errors compared to cascaded baselines.
The paper proposes Style-Debiased DPO (SD-DPO), a method for factuality-aware synthetic preference data to improve knowledge elicitation in large language models, addressing issues where standard DPO may incorrectly penalize correct responses due to stylistic differences.
StalePO introduces a token-level preference optimization method for machine translation that uses legacy post-edits to improve model performance, showing significant gains in quality metrics.
The paper proposes Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method for LLMs that uses comparison oracles to avoid likelihood displacement. It includes theoretical guarantees and experimental improvements over existing methods.
This paper proposes a budget-aware online teaching framework for web agents that reduces teacher calls and compute costs while maintaining performance.
Direct Diversity Optimization (DDO) is an offline post-training method that improves successful strategy coverage in LLM agents for sequential decision tasks, outperforming other methods in benchmarks like BabyAI, BabaIsAI, and WebShop.
Research from Stanford and Columbia shows that sycophantic agreement can emerge as an unintended consequence of contrastive preference optimization objectives, with teacher model bias transferring to student models even when preference data appears neutral across the dataset.
The paper introduces FiMI Banking, a sovereign conversational AI model for Indian retail banking, trained using preference optimization and reinforcement learning to enhance safe behavior and tool-use performance within regulatory constraints.