Tag
CoMerge is a conflict-driven preference optimization framework for merging multi-task LLMs, using self-supervised strategies to mitigate parameter interference and achieve high performance on benchmarks like MergeBench.
PLC-DPO enhances Direct Preference Optimization by routing noisy preference labels into clean, flipped, or tied cases using policy-reference margins, leading to improved performance across various benchmarks.
This paper proposes MoPLEx, an algorithm for learning mixtures of Plackett-Luce models to handle heterogeneous preferences in AI alignment, showing improved clustering and ranking accuracy over baselines.
This paper proposes Step-KTOder, a framework for code preference optimization that uses function-level execution feedback with unit tests to improve over outcome-only methods like KTO and DPO on benchmarks such as HumanEval+ and MBPP+.
Cross-lingual Ranking Preference Optimization (CRPO) is a novel framework that enhances multilingual LLM alignment by transferring English preference knowledge to target languages through hierarchical ranking optimization, demonstrating improved performance in instruction-following and knowledge utilization across multiple languages.
FAR-DPO is a feasibility-aware and robust direct preference optimization framework that enhances cyclic peptide design for drug discovery by aligning generative models with structural and biophysical constraints, improving success rates on benchmarks like PepGLAD and PepFlow.
The paper identifies manifold drift as a root cause of reward hacking in flow preference optimization and introduces ThermoDPO, a temperature-controlled method that anchors optimization on preferred samples to preserve data manifold integrity and improve performance.
Data-DPO is a target model-oriented data selection method for LLM supervised fine-tuning that learns data preferences through one-step probing and combines them with quality scores and diversity, outperforming baselines on Vision-Flan and LLaVA-CoT datasets.
This paper introduces SFS-DPO, a reinforcement learning two-stage framework for step-level self-verification and self-correction in LLMs, with a teacher-assisted variant SFS-DPO-R. It demonstrates improvements in self-correction effectiveness across multiple LLMs with less training data than prior approaches.
The paper proposes MAP-PO, a multi-agent framework that clusters annotators by labeling behavior and fine-tunes separate LLM agents per cluster using preference optimization, preserving disagreement in sexism detection tasks. Experiments on the EXIST 2024 dataset show that cluster-specific training is necessary and that a shared team-level reward keeps agents calibrated.
Soup is an open-source CLI that simplifies LLM fine-tuning and post-training with a single command, enabling QLoRA-based training on consumer GPUs with as little as 4 GB VRAM via layer streaming. The latest version adds preference losses like DPO, ORPO, SimPO, and KTO without doubling memory requirements.
This paper studies how preference optimization shapes LLM counselors' behavior in motivational interviewing, finding that penalizing confrontation trades goal persistence for relational attunement rather than teaching the balanced skill of rolling with resistance.
Presents TELLER, a dual-path iterative preference optimization approach for table entity linking, with direct-answer and reasoning paths that improve accuracy on TableInstruct and MammoTab V2 benchmarks.
LARA is a method for efficient adaptation that adds low-rank corrections to a frozen model's residual stream instead of modifying weights, matching LoRA's performance while enabling composable behaviors and inference-time steering.
This paper introduces DMAPO, a method for preference optimization that uses multi-evaluator consensus to select high-confidence on-policy responses, achieving strong alignment with significantly less data (only 3.45% acceptance rate) and outperforming baselines like SimPO on several benchmarks.
This paper demonstrates that the final window of pretraining significantly influences a model's response to post-training alignment, even when SFT performance is identical, suggesting that checkpoint evaluation should include the last training data.
Proposes Reliability-Guided Preference Optimization (RGPO) to handle inconsistent human feedback in LLM alignment by estimating annotator reliability and dynamically modulating training based on consensus, achieving superior performance over standard RLHF methods.
This paper proposes TD-DPO, a token-level difference-aware preference optimization method to mitigate sycophancy in LLMs for clinical autism intervention dialogue, achieving a better trade-off between sycophancy reduction and intervention ability retention.
This paper introduces a regularization technique for Direct Alignment Algorithms (DAAs) that maintains normalized response probabilities, mitigating over-optimization and likelihood displacement. The method improves generation quality and benchmark performance, achieving over 20% relative increase on AlpacaEval2 and 9% gains on general benchmarks for Llama-3.1-8B-Instruct.
This paper proposes RIMS, a three-stage preference optimization framework for small-scale language models in retrieval-augmented generation, using synthetic chain-of-thought data and a differentiable soft aggregation mechanism to improve robustness against noisy evidence. Experiments show consistent gains over baselines on multi-hop QA benchmarks.