GroupDPO: Memory efficient Group-wise Direct Preference Optimization

# GroupDPO: Memory efficient Group-wise Direct Preference Optimization
Source: https://arxiv.org/html/2604.15602
Jixuan Leng CMU [email protected] &Si Si Google Deepmind [email protected] Hsiang-Fu Yu Google [email protected] &Vinod Raman Google Deepmind [email protected] &Inderjit S. Dhillon Google [email protected]

###### Abstract

Preference optimization is widely used to align Large Language Models (LLMs) with preference feedback. However, most existing methods train on a single positive-negative pair per prompt, discarding additional supervision available in preference datasets that typically contain multiple candidate responses. Motivated by this limitation, recent work explores group-wise preference optimization, which jointly contrasts multiple responses for the same prompt, but its empirical behavior and scalability remain underexplored due to the memory overhead of group-coupled objectives. In this work, we introduce a memory-efficient group-wise preference optimization algorithm that preserves gradients while decoupling samples during backpropagation, substantially reducing peak memory usage, which enables scalable training with larger group sizes. Across both offline and online alignment settings, we show that leveraging multiple responses consistently outperforms single-pair training. Furthermore, incorporating a negative log-likelihood (NLL) term on positive responses is critical for both performance gains and training stability.

GroupDPO: Memory efficient Group-wise Direct Preference Optimization

Jixuan Leng††thanks:Work done while interning at [email protected] SiGoogle [email protected]

Hsiang-Fu [email protected] RamanGoogle [email protected] S. [email protected]

## 1 Introduction

Refer to captionFigure 1:Overview of Group DPO and our memory-efficient surrogate implementation. Top: Standard DPO trains on a single positive-negative pair, while group-wise objectives leverage multiple responses but require joint forward and backward passes, leading to high memory usage and OOM with large group sizes. Bottom: Our surrogate avoids group-coupled backpropagation by precomputing per-sample coefficients in a no-gradient pass and optimizing a sample-level loss, substantially reducing peak memory overhead and enabling scalable training with larger groups.Preference optimization has become a central component of modern large language model (LLM) alignment. Rather than relying solely on supervised fine-tuning, many alignment pipelines train policy models using preference feedback that compares alternative responses to the same prompt. Early approaches relied on Reinforcement Learning from Human Feedback (RLHF)(Christiano et al., 2017; Ouyang et al., 2022; Yang et al., 2025b), where a reward model is first trained then used to optimize the policy model. More recently, Direct Preference Optimization (DPO)(Rafailov et al., 2023) and related methods(Meng et al., 2024; Hong et al., 2024; Liu et al., 2024b) simplify this pipeline by directly optimizing the policy from preference data, bypassing reward-model training and policy-gradient RL.

Despite these advances, most preference optimization methods operate on *pairwise* comparisons between a single positive and negative response. In practice, however, preference datasets are often constructed by sampling multiple candidate responses for each prompt and evaluating them using human or automated feedback(Cui et al., 2023). To apply pairwise objectives such as DPO, these response sets are typically *binarized* by selecting a single positive and negative response while discarding the remaining candidates. This reduction loses potentially useful supervision signals about the relative quality of responses within the group. As a result, recent work has begun exploring *group-wise* or *listwise* preference learning methods that train on response sets rather than individual pairs.

Group-wise preference optimization can provide richer supervision by jointly contrasting multiple candidate responses for the same prompt. Several recent methods extend pairwise preference learning to this setting using ranking or set-level objectives(Song et al., 2024; Gupta et al., 2024; Liu et al., 2025b; Chen et al., 2024a). However, the practical behavior of these methods remains poorly understood, as prior work typically proposes individual objectives and evaluates them in isolation, making systematic comparisons under consistent training settings difficult. Moreover, group-wise objectives introduce cross-response dependencies within each prompt group. Because the loss depends on interactions between positive and negative responses, gradients become coupled across samples and require constructing a joint computation graph over the group. In naive implementations(von Werra et al., 2020; Gupta et al., 2024), activations for all samples must therefore be retained until backpropagation, causing memory usage to grow exponentially with group size. Consequently, memory costs restrict prior work to small groups, limiting exploitation of group-level supervision.

In this work, we study group-wise preference optimization from both practical and empirical perspectives. To address the computational challenges of group-wise objectives, we introduce a memory-efficient surrogate implementation that reformulates the objective into a sample-level surrogate loss with matched first-order gradients. The key idea is to compute per-sample gradient coefficients using an extra lightweight no-gradient pass and then perform standard token-level backpropagation with these coefficients. This decouples gradients across samples and eliminates the need to retain activations for samples simultaneously, making memory overhead largely independent of the group size.

Using this implementation, we conduct an empirical study of several representative group-wise objectives under a unified formulation across both offline and online alignment settings. Our results reveal two key findings. First, incorporating multiple samples per prompt consistently outperforms single-pair training, indicating that group-level supervision provides a richer learning signal. Second, group-wise training can be unstable: including an additional negative log-likelihood (NLL) term(Liu et al., 2024b; Pang et al., 2024; Wang et al., 2024a; Grattafiori et al., 2024; Pal et al., 2024) over positive samples plays a crucial role in preventing training collapse and improving the final performance.

We evaluate our approach across multiple models and training settings. Results on multi-domain benchmarks show that group-wise preference optimization consistently outperforms single-pair training. Meanwhile, the proposed surrogate implementation significantly reduces memory usage while maintaining competitive training latency, making group-wise training more practical at larger scales.

Our contributions are summarized as follows:

- We provide a controlled comparison of group-wise variants under offline and online settings.
- We show that training with groups of responses improves performance over pairwise training, and that positive-response NLL regularization is key to stabilizing group-wise optimization.
- We introduce a memory-efficient surrogate that is first-order equivalent to vanilla implementation.
- We show that this surrogate substantially reduces peak memory overhead and enables training with larger groups while keeping favorable efficiency.

## 2 Related Work

Preference Alignment. Aligning LLMs with human preferences has relied on RLHF(Christiano et al., 2017; Ouyang et al., 2022), where a reward model trained on preference comparisons is optimized using policy gradient methods such as Proximal Policy Optimization (PPO)(Schulman et al., 2017). While effective, RLHF introduces computational overhead and training instability due to extra reward-model training and reinforcement learning.

DPO(Rafailov et al., 2023) provides a simpler alternative by reformulating preference alignment as a classification problem over chosen and rejected responses, allowing the policy to be optimized directly without training an explicit reward model. Building on this formulation, subsequent work has explored extensions to improve training stability and robustness. One direction augments the objective with supervised learning signals to mitigate issues such as likelihood degradation and over-optimization. For example, Regularized Preference Optimization (RPO)(Liu et al., 2024b) introduces a negative log-likelihood (NLL) term over chosen responses as an implicit regularizer, while Iterative Reasoning Preference Optimization (IRPO)(Pang et al., 2024) adopts a similar hybrid objective to stabilize online alignment for complex reasoning tasks. Another line of research investigates biases in preference learning, including length exploitation and overconfidence, and proposes calibration and normalization methods to address them(Park et al., 2024; Liu et al., 2024a; Leng et al., 2024).

Because DPO relies on a frozen reference model, several works have also explored reference-free formulations that simplify training. For example, SimPO(Meng et al., 2024) removes the reference model and constructs the preference objective using length-normalized log-likelihoods of the policy, while ORPO(Hong et al., 2024) integrates preference optimization directly into supervised fine-tuning through an odds-ratio objective. Beyond human-labeled comparisons, recent self-improvement approaches such as SPIN(Chen et al., 2024b) leverage iterative self-play to generate extra preference data for improvement during training.

Group-wise Preference Learning. Although DPO operates on pairwise comparisons, modern preference datasets often contain multiple candidate responses for each prompt. Reducing these response sets to independent pairs discards useful relative information and limits the supervision signal available during training. To address this limitation, recent work has increasingly explored group-wise or listwise preference learning, drawing connections to classical learning-to-rank methods.

Several approaches extend preference optimization beyond pairwise comparisons using ranking-based objectives. Preference Ranking Optimization (PRO)(Song et al., 2024) and Listwise Preference Optimization (LiPO)(Liu et al., 2025b) generalize pairwise comparisons to full rankings using the Plackett-Luce model(Plackett, 1975). Multi-Preference Optimization (MPO)(Gupta et al., 2024) instead models preferences using a contrastive objective defined over the response groups. Similar group-wise or listwise learning paradigms have also been explored in recommendation systems, where methods such as S-DPO(Chen et al., 2024a), LPO4Rec(Li et al., 2025), and RankGR(Fu et al., 2026) leverage large candidate sets to improve ranking and retrieval performance.

Despite these advances, prior work on group-wise preference optimization has two key limitations. First, algorithms are often proposed for different settings, lacking a unified comparison of group-wise objectives. Second, many methods operate on relatively small response groups due to the memory overhead of group-wise losses. We address these limitations by systematically comparing representative methods under a unified experimental framework, studying larger group sizes, and introducing a memory-efficient surrogate implementation whose GPU memory usage does not scale with group size, enabling scalable training.

## 3 Background

Group DPO Setup. We consider a preference dataset where responses are organized into prompt-specific groups. Each group g corresponds to a prompt x and contains positive responses P_g and negative responses N_g. Responses within each group are treated as unordered, with no intra-set ranking. This setting arises naturally in practice: for example, in online training with rule-based outcome rewards (e.g., math reasoning), responses are often partitioned only by correctness, and in recommendation systems feedback typically indicates preferred vs. non-preferred items without ordering.

Implicit preference score. Following DPO, we define the implicit preference score as follows:

u_θ(y|x) = β(log π_θ(y|x) − log π_ref(y|x)),

a scaled log-likelihood ratio between the policy π_θ and reference model π_ref, and β is a scaling factor.

A Unified Group DPO objective view. A broad class of preference objectives can be written as

ℒ_group(θ) = (1/G) ∑_{g=1}^G φ_g(u_{P_g}(θ), u_{N_g}(θ))

where u_{P_g} = {u_θ(y|x) : y ∈ P_g} and similarly for u_{N_g}, and φ_g is method-specific, differing mainly in how scores are aggregated or contrasted across groups. Despite differences in loss definitions, these objectives share a common computational issue: cross-response coupling within each group, which makes direct optimization memory-intensive. To address this, we introduce a memory-efficient and gradient-equivalent surrogate loss in Section 4.

## 4 Memory-Efficient Surrogate

Motivation: cross-response dependencies in group-wise objectives. Group-wise preference objectives couple responses within each prompt group. Many variants rely on cross-response interactions (e.g., score difference between positive and negative responses u_p − u_n or softmax-based aggregations such as log ∑ exp(u_n)), meaning that the gradient contribution of each response depends on the scores of other responses in the same group. As a result, the gradient for a single response cannot be computed