Contrastive Weak-to-strong Generalization

# Contrastive Weak-to-strong Generalization
Source: [https://arxiv.org/html/2510.07884](https://arxiv.org/html/2510.07884)
Junfeng FangJiaxin WuTianyu ZhangChen GaoXiang WangXiangnan HeYang Deng

###### Abstract

Weak\-to\-strong generalization provides a promising paradigm for scaling large language models \(LLMs\) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling\. However, its robustness and generalization are hindered by the noise and biases in weak\-model outputs, which limit its applicability in practice\. To address this challenge, we leverage implicit rewards, which approximate explicit rewards through log\-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding \(CD\), a decoding strategy shown to reduce noise in LLM generation\. Building on this connection, we proposeContrastive Weak\-to\-Strong Generalization \(ConG\), a framework that employs contrastive decoding between pre\- and post\-alignment weak models to generate higher\-quality samples\. This approach enables more reliable capability transfer, denoising, and improved robustness, substantially mitigating the limitations of traditional weak\-to\-strong methods\. Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG\. Taken together, our findings highlight the potential of ConG to advance weak\-to\-strong generalization and provide a promising pathway toward AGI\. Our code is available at:[https://github\.com/jianghoucheng/ConG](https://github.com/jianghoucheng/ConG)

Machine Learning, ICML

## 1Introduction

Weak\-to\-strong generalization has emerged as a promising paradigm for scaling the capabilities of large language models \(LLMs\)\(Burnset al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib11); Yaoet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib16); Liet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib34); Somerstepet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib35); Zhouet al\.,[2024b](https://arxiv.org/html/2510.07884#bib.bib28)\)\. By leveraging supervised samples generated from an aligned weaker model, a stronger model can be directly trained without requiring additional reward modeling or human feedback\(Burnset al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib11); Ouyanget al\.,[2022](https://arxiv.org/html/2510.07884#bib.bib9); Leeet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib13)\)\. This enables LLMs to transfer and extend capabilities to even stronger models, providing opportunities for self\-enhancement and thus offering a potential pathway toward Artificial General Intelligence \(AGI\)\(Goertzel,[2014](https://arxiv.org/html/2510.07884#bib.bib53)\)\.

Despite encouraging progress, the paradigm suffers from poor robustness and limited generalization\(Yaoet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib16); Yanget al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib38)\)\. We attribute this to the inherent biases and preferences embedded in the weaker model: the samples it generates often contain noise and are of relatively low quality\. As a result, the stronger model fails to generalize reliably, thereby restricting the applicability of weak\-to\-strong methods\(Lyuet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib31)\)\. This raises a central research question:How can we extract higher\-quality samples from weak models, without relying on explicit rewards \(e\.g\., human feedback or reward models\), to achieve more effective weak\-to\-strong generalization?

![Refer to caption](https://arxiv.org/html/2510.07884v2/figures/intro.png)Figure 1:Overview of our proposed ConG\. \(a\) Paradigm illustration comparing traditional weak\-to\-strong methods with ConG\. \(b\) Scatter plot showing the correlation between implicit and explicit rewards, together with a comparison of sample rewards from naive sampling and contrastive decoding\. \(c\) Radar chart comparing weak\-to\-strong methods on AlpacaEval2 \(AE\) and Arena\-Hard \(AH\); metrics with underlines denote Qwen2\.5\-7B\-Instruct, while those without underlines correspond to Llama3\-8B\-Instruct\. Best viewed in color\.Recent success of implicit rewards in preference alignment and reasoning enhancement motivates our approach\(Yuanet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib36); Cuiet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib37)\)\. Specifically, implicit reward parameterizes the reward as the log\-likelihood ratio between outputs from the post\-alignment and pre\-alignment models, and prior work has shown it to be an unbiased approximation of explicit reward\(Rafailovet al\.,[2023](https://arxiv.org/html/2510.07884#bib.bib10)\)\. This suggests that implicit reward can serve as a reliable signal for assessing sample quality\. Moreover, we observe that its log\-ratio structure closely matches the form ofContrastive Decoding \(CD\), a recently proposed decoding strategy proven to mitigate noise in LLM generation\(Liet al\.,[2023](https://arxiv.org/html/2510.07884#bib.bib17)\)\. This structural consistency implies that theCD process can be interpreted as generating responses that maximize implicit reward\. We formalize this conclusion as theCD–Implicit Reward Correlation\. Empirically, Figure[1](https://arxiv.org/html/2510.07884#S1.F1)\(b\) shows a strong linear correlation between implicit and explicit rewards, with CD\-generated samples concentrating in the high implicit\-reward region, consistent with this correlation\.

The CD–Implicit Reward Correlation provides a practical pathway for generating higher\-quality samples from weak models to train stronger ones, enabling weak\-to\-strong generalization with reduced noise risk\. As illustrated in Figure[1](https://arxiv.org/html/2510.07884#S1.F1)\(a\), instead of directly using samples generated by the aligned weak model, we employ contrastive decoding between the pre\-alignment and post\-alignment weak models to generate training samples for the strong model\. Theoretically, samples obtained via contrastive decoding preserve the full signal of target preferences and approximately maximize implicit reward, thereby supporting effective weak\-to\-strong generalization\. We refer to this simple yet effect paradigm asContrastive Weak\-to\-Strong Generalization \(ConG\)\. In addition to weak\-to\-strong generalization, ConG can naturally reduce to*self\-alignment*when the weak and strong models are instantiated as the same model, further broadening its applicability\.

We evaluate our approach on two mainstream LLM families—Qwen2\.5\(Yanget al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib25)\)and Llama3\(Dubeyet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib24)\)\. All models are trained for both weak\-to\-strong alignment and self\-alignment using the UltraFeedback dataset\(Cuiet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib30)\), and evaluated on AlpacaEval2\(Duboiset al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib22)\)and Arena\-Hard\(Liet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib23)\)\. Experimental results demonstrate that ConG consistently and significantly outperforms traditional weak\-to\-strong methods across all models\. On average, it yields a gain of about 16\.5% over the base model, as shown in Figure[1](https://arxiv.org/html/2510.07884#S1.F1)\(c\)\. These results confirm the generality of ConG across different alignment scenarios and highlight its ability to improve capability transfer, denoising, and robustness, offering a promising pathway toward AGI\.

## 2Preliminary

### 2\.1Reinforcement Learning from Human Feedback

In preference alignment, the training dataset𝒟=\{\(x,yw,yl\)\}\\mathcal\{D\}=\\\{\(x,y\_\{w\},y\_\{l\}\)\\\}consists of promptsxxpaired with two candidate responses, whereyw≻yl∣xy\_\{w\}\\succ y\_\{l\}\\mid xindicates thatywy\_\{w\}is preferred overyly\_\{l\}\. A common assumption is that preferences follow the Bradley–Terry model\(Bradley and Terry,[1952](https://arxiv.org/html/2510.07884#bib.bib26)\), in which the probability of preferringywy\_\{w\}overyly\_\{l\}is proportional to the exponentiated reward\. Under this assumption, a reward modelrθ​\(x,y\)r\_\{\\theta\}\(x,y\)is trained by maximizing the pairwise log\-likelihood:

ℒRM​\(rθ\)=−𝔼\(x,yw,yl\)∼𝒟​\[log⁡σ​\(rθ​\(x,yw\)−rθ​\(x,yl\)\)\],\\mathcal\{L\}\_\{\\mathrm\{RM\}\}\(r\_\{\\theta\}\)=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\\left\[\\log\\sigma\\big\(r\_\{\\theta\}\(x,y\_\{w\}\)\-r\_\{\\theta\}\(x,y\_\{l\}\)\\big\)\\right\],\(1\)whereσ​\(⋅\)\\sigma\(\\cdot\)denotes the sigmoid function\. Once the reward model is trained, the language model is treated as a policyπθ​\(y∣x\)\\pi\_\{\\theta\}\(y\\mid x\)and optimized to maximize the expected reward while remaining close to a reference policyπref\\pi\_\{\\mathrm\{ref\}\}:

maxπθ𝔼x∼𝒟,y∼πθ\(⋅\|x\)\[rθ\(x,y\)\]−βKL\(πθ\(⋅\|x\)∥πref\(⋅\|x\)\),\\max\_\{\\pi\_\{\\theta\}\}\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},y\\sim\\pi\_\{\\theta\}\(\\cdot\|x\)\}\[r\_\{\\theta\}\(x,y\)\]\-\\beta\\mathrm\{KL\}\\big\(\\pi\_\{\\theta\}\(\\cdot\|x\)\\\|\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x\)\\big\),\(2\)whereβ\>0\\beta\>0controls the regularization strength\. This objective is typically optimized using Proximal Policy Optimization \(PPO\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2510.07884#bib.bib27); Ouyanget al\.,[2022](https://arxiv.org/html/2510.07884#bib.bib9); Christianoet al\.,[2017](https://arxiv.org/html/2510.07884#bib.bib8)\)\.

### 2\.2Direct Preference Optimization \(DPO\)

Direct Preference Optimization \(DPO\)\(Rafailovet al\.,[2023](https://arxiv.org/html/2510.07884#bib.bib10)\)provides a stable, scalable alternative to RLHF by directly optimizing the policy from preference comparisons, without an explicit reward model or on\-policy sampling\. DPO can be derived by reparameterizing the reward functionr​\(x,y\)r\(x,y\)through the closed\-form solution to the KL\-regularized reward maximization problem:

r​\(x,y\)=β​log⁡πr​\(y∣x\)πref​\(y∣x\)\+β​log⁡Z​\(x\),r\(x,y\)\\;=\\;\\beta\\log\\frac\{\\pi\_\{r\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\\;\+\\;\\beta\\log Z\(x\),\(3\)whereπr\\pi\_\{r\}is the aligned policy,πref\\pi\_\{\\mathrm\{ref\}\}is the reference policy, andZ​\(x\)Z\(x\)is a partition function independent ofyy\. The corresponding*implicit reward*for a trainable policyπθ\\pi\_\{\\theta\}is

r^​\(x,y\)=β​log⁡πθ​\(y∣x\)πref​\(y∣x\)\.\\hat\{r\}\(x,y\)\\;=\\;\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\.\(4\)Given the preference dataset𝒟=\{\(x,yw,yl\)\}\\mathcal\{D\}=\\\{\(x,y\_\{w\},y\_\{l\}\)\\\}\(withyw≻yl∣xy\_\{w\}\\succ y\_\{l\}\\mid x\), substitutingr^\\hat\{r\}into the pairwise likelihood yields the DPO loss:

ℒDPO​\(πθ\)=−𝔼\(x,yw,yl\)∼𝒟​\[log⁡σ​\(r^​\(x,yw\)−r^​\(x,yl\)\)\]\.\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}\(\\pi\_\{\\theta\}\)\\;=\\;\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\\Big\[\\log\\sigma\\\!\\big\(\\hat\{r\}\(x,y\_\{w\}\)\-\\hat\{r\}\(x,y\_\{l\}\)\\big\)\\Big\]\.\(5\)

## 3CD–Implicit Reward Correlation

In this section, we begin by discussing how language models can be interpreted as implicit rewards that capture preferences \(Section[3\.1](https://arxiv.org/html/2510.07884#S3.SS1)\)\. Building on this foundation, Section[3\.2](https://arxiv.org/html/2510.07884#S3.SS2)demonstrates that the formulation of implicit rewards is mathematically consistent with contrastive decoding, leading to the formalization of the CD–Implicit Reward Correlation\. Finally, Section[3\.3](https://arxiv.org/html/2510.07884#S3.SS3)presents empirical evidence showing that contrastive decoding outputs inherently encode preference information and provide higher\-quality supervision signals, thereby laying the foundation for the weak\-to\-strong generalization method introduced in the next section\.

### 3\.1Language Models as Reward Functions

As defined in Eqn\.[4](https://arxiv.org/html/2510.07884#S2.E4), the implicit rewardr^​\(x,y\)\\hat\{r\}\(x,y\)can be expressed as the difference in log\-probabilities between the aligned policyπr\\pi\_\{r\}and the reference policyπref\\pi\_\{\\text\{ref\}\}\. By factorizing the log\-probability at the token level\(Zhouet al\.,[2024b](https://arxiv.org/html/2510.07884#bib.bib28)\), we obtain:

r^​\(x,y\)=β​∑t=1\|y\|\[log⁡πθ​\(yt∣x,y<t\)−log⁡πref​\(yt∣x,y<t\)\],\\hat\{r\}\(x,y\)=\\beta\\sum\_\{t=1\}^\{\|y\|\}\\left\[\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\-\\log\\pi\_\{\\text\{ref\}\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\\right\],\(6\)wherey<ty\_\{<t\}denotes the prefix of the firstt−1t\-1tokens ofyy, andyty\_\{t\}denotes the token at positiontt\. This token\-level decomposition shows that the implicit reward can be viewed as the sum of per\-token log\-probability differences between the aligned policyπr\\pi\_\{r\}and the reference policyπref\\pi\_\{\\text\{ref\}\}\. In practice, this token\-level implicit reward serves as a dense, model\-based proxy for the explicit rewardr​\(x,y\)r\(x,y\)provided by a trained reward model, enabling preference estimation directly from model behavior without additional annotation\.

### 3\.2Connection between Contrastive Decoding and Implicit Reward

As shown in Section[3\.1](https://arxiv.org/html/2510.07884#S3.SS1), the token\-level implicit reward measures the relative preference between two policiesπr\\pi\_\{r\}andπref\\pi\_\{\\text\{ref\}\}via the per\-token log\-probability difference\. This formulation is mathematically consistent with the inference\-time decoding strategy–contrastive decoding\(Liet al\.,[2023](https://arxiv.org/html/2510.07884#bib.bib17)\), which generates tokens by comparing their probability distribution underπr\\pi\_\{r\}andπref\\pi\_\{\\text\{ref\}\}\.

At each decoding steptt, contrastive decoding defines the next\-token distribution as:

pα​\(yt∣x,y<t\)=softmax​\(ℱ​\(yt\)\),\\displaystyle p\_\{\\alpha\}\(y\_\{t\}\\mid x,y\_\{<t\}\)=\\mathrm\{softmax\}\\big\(\\mathcal\{F\}\(y\_\{t\}\)\\big\),\(7\)ℱ​\(yt\)=\{g​\(yt\),if​yt∈𝒱head−∞,otherwise\\displaystyle\\mathcal\{F\}\(y\_\{t\}\)=where​g​\(yt\)=\(1−α\)​log⁡πr​\(yt∣x,y<t\)πref​\(yt∣x,y<t\)\\displaystyle\\text\{where \}g\(y\_\{t\}\)=\(1\-\\alpha\)\\log\\frac\{\\pi\_\{r\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\+α​log⁡πr​\(yt∣x,y<t\)\.\\displaystyle\\qquad\\qquad\\quad\+\\alpha\\log\\pi\_\{r\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\.whereα∈\[0,1\]\\alpha\\in\[0,1\]\(contrastive coefficient\) controls the relative weight of the contrastive term\. Smallerα\\alphaincreases the influence of the probability gap, biasing the model toward higher\-implicit\-reward tokens, whileα=1\\alpha=1reduces to standard decoding underπr\\pi\_\{r\}\.

The candidate set𝒱head\\mathcal\{V\}\_\{\\text\{head\}\}is obtained via vocabulary pruning:

𝒱head\(x,y<t\)=\{yt∈𝒳:πr\(yt∣x,y<t\)≥\\displaystyle\\mathcal\{V\}\_\{\\text\{head\}\}\(x,y\_\{<t\}\)=\\big\\\{y\_\{t\}\\in\\mathcal\{X\}:\\pi\_\{r\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\\geq\(8\)λ⋅maxw⁡πr​\(w∣x,y<t\)\\displaystyle\\lambda\\cdot\\max\_\{w\}\\pi\_\{r\}\(w\\mid x,y\_\{<t\}\)\},\\displaystyle\\big\\\},whereλ∈\[0,1\]\\lambda\\in\[0,1\]is the threshold\. If the predicted probability of a token underπr\\pi\_\{r\}is far smaller than the top candidate in the same decoding step, it is unlikely to be a reasonable prediction; thus, such tokens are excluded from𝒱head\\mathcal\{V\}\_\{\\text\{head\}\}by setting their logits to−∞\-\\inftyin Eqn\.[7](https://arxiv.org/html/2510.07884#S3.E7)\.

Comparing Eqn\.[7](https://arxiv.org/html/2510.07884#S3.E7)with the token\-level implicit reward in Eqn\.[4](https://arxiv.org/html/2510.07884#S2.E4)shows that the contrastive termlog⁡πr​\(yt∣x,y<t\)πref​\(yt∣x,y<t\)\\log\\frac\{\\pi\_\{r\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}is exactly the implicit reward up to a scaling factor\(1−α\)\(1\-\\alpha\)\. Therefore, under the contrastive decoding distributionpαp\_\{\\alpha\}, the decoding objective is to findy∗y^\{\\ast\}that approximately maximizes the implicit reward, as follows:

y∗=argmaxy∑t=1\|y\|\[\\displaystyle y^\{\\ast\}=\\arg\\max\_\{y\}\\sum\_\{t=1\}^\{\|y\|\}\\bigg\[\(1−α\)​log⁡πr​\(yt∣x,y<t\)πref​\(yt∣x,y<t\)\\displaystyle\(1\-\\alpha\)\\log\\frac\{\\pi\_\{r\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\(9\)\+αlogπr\(yt∣x,y<t\)\]\.\\displaystyle\+\\alpha\\log\\pi\_\{r\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\\bigg\]\.Based on this derivation, samples generated with contrastive decoding are expected to have higher implicit rewards than standard decoding, andα\\alphaoffers a direct control over the implicit reward level by adjusting the weight of the contrastive term \(see Appendix[C](https://arxiv.org/html/2510.07884#A3)for a detailed proof\)\. We formalize this conclusion as theCD–Implicit Reward Correlation\.

### 3\.3Empirical Analysis

![Refer to caption](https://arxiv.org/html/2510.07884v2/figures/method_1.png)Figure 2:Performance comparison of contrastive decoding with different contrastive coefficientsα\\alphafor Llama3\.2\-3B\-Instruct \(top row\) and Qwen2\.5\-3B\-Instruct \(bottom row\)\. \(a,d\) Relationship betweenα\\alphaand implicit/explicit rewards\. \(b,e\) Relationship betweenα\\alphaand response length\. \(c,f\) Win\-rate matrices showing the proportion of cases where rowα\\alphaoutperforms columnα\\alpha\.In Section[3\.2](https://arxiv.org/html/2510.07884#S3.SS2), we established the theoretical correspondence between contrastive decoding and implicit rewards\. We empirically examine this connection by varying the contrastive coefficientα\\alphaand analyzing its effect on both rewards and generation behavior\.

We first consider howα\\alphainfluences the implicit and explicit rewards\. The results, summarized in Figure[2](https://arxiv.org/html/2510.07884#S3.F2)\(a\) and Figure[2](https://arxiv.org/html/2510.07884#S3.F2)\(d\), reveal a clear pattern: implicit reward decreases monotonically with increasingα\\alpha, consistent with our theoretical prediction, while explicit reward remains relatively high forα∈\[0,0\.5\]\\alpha\\in\[0,0\.5\]before dropping sharply onceα\>0\.5\\alpha\>0\.5\. This suggests that smallerα\\alphavalues preserve more preference information fromπr\\pi\_\{r\}, whereas larger values overemphasize the contrastive term and deteriorate overall generation quality\.

To ensure that these differences are not confounded by superficial factors such as response length, we analyze the length distributions acrossα\\alphain Figure[2](https://arxiv.org/html/2510.07884#S3.F2)\(b\) and Figure[2](https://arxiv.org/html/2510.07884#S3.F2)\(e\)\. The distributions remain stable regardless ofα\\alpha, indicating that the observed performance variation arises from content quality rather than response length bias\.

Finally, the relative performance across differentα\\alphasettings is captured by the win\-rate matrices in Figure[2](https://arxiv.org/html/2510.07884#S3.F2)\(c\) and Figure[2](https://arxiv.org/html/2510.07884#S3.F2)\(f\)\. Generations withα∈\[0,0\.5\]\\alpha\\in\[0,0\.5\]consistently outperform those withα∈\[0\.6,1\.0\]\\alpha\\in\[0\.6,1\.0\], reinforcing the observation that moderate contrastive strength leads to higher\-quality outputs\. Taken together, these results provide strong empirical support for our theoretical analysis, highlighting the critical role ofα\\alphain balancing reward alignment and generation robustness\.

## 4Contrastive Weak\-to\-Strong Generalization \(ConG\)

Building on the CD–Implicit Reward Correlation established in Section[3\.3](https://arxiv.org/html/2510.07884#S3.SS3), we introduce Contrastive Weak\-to\-Strong Generalization \(ConG\)\. The central idea is to leverage contrastive decoding \(CD\) to extract higher\-quality responses from weak models, and use them to drive the generalization of stronger models\. ConG consists of two stages: \(i\) ConG\-S, which employs CD responses for SFT to provide a high\-reward initialization, and \(ii\) ConG, which further strengthens weak\-to\-strong generalization with DPO\.

Stage I: ConG\-S \(Contrastive Decoding for SFT\)\.Letπrw\\pi\_\{r\}^\{\\mathrm\{w\}\}denote the post\-alignment weak model andπrefw\\pi\_\{\\mathrm\{ref\}\}^\{\\mathrm\{w\}\}the pre\-alignment weak model\. For each promptx∈𝒳x\\in\\mathcal\{X\}, we instantiate contrastive decoding with the weak models to obtain the decoding distribution defined in Eqn\.[8](https://arxiv.org/html/2510.07884#S3.E8)\. Decoding underpαp\_\{\\alpha\}yields a chosen sampleywy\_\{w\}for eachxx, forming𝒟SFT=\{\(x,yw\)\}\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}=\\\{\(x,y\_\{w\}\)\\\}\. Letπrefs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{ref\}\}be the initial strong model; we obtainπSFTs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}by minimizing the standard SFT loss on chosen samples:

ℒSFT​\(πθs\)=−𝔼\(x,yw\)∼𝒟SFT​\[log⁡πθs​\(yw∣x\)\]\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\\\!\\left\(\\pi^\{\\mathrm\{s\}\}\_\{\\theta\}\\right\)=\-\\mathbb\{E\}\_\{\(x,y\_\{w\}\)\\sim\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}\}\\left\[\\log\\pi^\{\\mathrm\{s\}\}\_\{\\theta\}\(y\_\{w\}\\mid x\)\\right\]\.\(10\)This stage moves the strong model’s policy toward the CD–induced preference distribution, providing a high\-reward starting point for preference optimization\.

Stage II: ConG \(Generalization with DPO\)\.After SFT, we further refine the strong model using DPO\. For each promptxx, we sample an additional responseyly\_\{l\}fromπSFTs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}under standard decoding and pair it with the corresponding CD responseywy\_\{w\}\. This construction is justified by two factors: \(i\) CD responses approximate maximizers of implicit reward \(Eqn\.[9](https://arxiv.org/html/2510.07884#S3.E9)\), so in expectation their implicit reward satisfiesr^w\>r^l\\hat\{r\}\_\{w\}\>\\hat\{r\}\_\{l\}\(see Appendix[C](https://arxiv.org/html/2510.07884#A3)for a detailed proof\)\. \(ii\) sinceπSFTs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}has been trained onywy\_\{w\}, the distributions ofywy\_\{w\}andyly\_\{l\}are well matched, ensuring that their comparison isolates reward differences rather than distributional shifts\. We then optimize the strong model with the DPO loss, takingπSFTs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}as the reference policy:

ℒDPO\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}\(πθs\)=−𝔼\(x,yw,yl\)∼𝒟DPO\[logσ\(\\displaystyle\(\\pi^\{\\mathrm\{s\}\}\_\{\\theta\}\)=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\_\{\\mathrm\{DPO\}\}\}\\bigg\[\\log\\sigma\\bigg\(\(11\)βlogπθs​\(yw∣x\)πSFTs​\(yw∣x\)−βlogπθs​\(yl∣x\)πSFTs​\(yl∣x\)\)\]\\displaystyle\\beta\\log\\frac\{\\pi^\{\\mathrm\{s\}\}\_\{\\theta\}\(y\_\{w\}\\mid x\)\}\{\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}\(y\_\{w\}\\mid x\)\}\-\\beta\\log\\frac\{\\pi^\{\\mathrm\{s\}\}\_\{\\theta\}\(y\_\{l\}\\mid x\)\}\{\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}\(y\_\{l\}\\mid x\)\}\\bigg\)\\bigg\]whereβ\>0\\beta\>0controls preference sharpness andσ​\(⋅\)\\sigma\(\\cdot\)denotes the logistic function\. This stage exploits the reward gapr^w\>r^l\\hat\{r\}\_\{w\}\>\\hat\{r\}\_\{l\}while maintaining distributional consistency, thereby pushing the strong model toward more reliable and robust generalization\.

Together, ConG\-S and ConG constitute our framework for contrastive weak\-to\-strong generalization\. A detailed algorithmic description is provided in Appendix[A](https://arxiv.org/html/2510.07884#A1)\. Notably, whenπrefw=πrefs\\pi^\{\\mathrm\{w\}\}\_\{\\mathrm\{ref\}\}=\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{ref\}\}, ConG naturally reduces to a form of*self\-alignment*\.

## 5Experiments

Table 1:Main results on AlpacaEval2 and Arena\-Hard\.“Base” denotes the reference model without additional alignment\. “WR” denotes the raw win rate, “LC” the length\-controlled win rate, “SC” the style\-controlled win rate, and “Avg\.” the average score across benchmarks\. Best results are highlighted in bold and second\-best are underlined\.Qwen2\.5\-3B\-Instruct \(Weak\)Llama3\.2\-3B\-Instruct \(Weak\)MethodAlpacaEval 2Arena\-HardAvg\.AlpacaEval 2Arena\-HardAvg\.LCWRSCWRLCWRSCWRBase13\.8±\\pm0\.314\.3±\\pm1\.530\.5±\\pm2\.333\.8±\\pm2\.722\.820\.2±\\pm0\.423\.8±\\pm1\.422\.6±\\pm2\.220\.2±\\pm2\.621\.8DPO29\.4±\\pm0\.434\.9±\\pm1\.542\.4±\\pm2\.544\.3±\\pm2\.737\.831\.4±\\pm0\.334\.8±\\pm1\.729\.8±\\pm2\.129\.3±\\pm2\.429\.6ORPO22\.7±\\pm0\.226\.3±\\pm1\.434\.5±\\pm2\.138\.0±\\pm2\.830\.427\.7±\\pm0\.530\.1±\\pm1\.524\.7±\\pm2\.425\.5±\\pm2\.225\.8SimPO34\.1±\\pm0\.335\.9±\\pm1\.345\.7±\\pm2\.850\.1±\\pm2\.941\.534\.0±\\pm0\.436\.9±\\pm1\.432\.0±\\pm2\.531\.2±\\pm2\.333\.5ConG\-S\(self\)33\.3±\\pm0\.437\.7±\\pm1\.746\.6±\\pm2\.351\.8±\\pm2\.742\.430\.9±\\pm0\.333\.0±\\pm1\.633\.3±\\pm2\.831\.6±\\pm2\.232\.2ConG\(self\)35\.9±\\pm0\.543\.3±\\pm1\.649\.2±\\pm2\.453\.5±\\pm2\.945\.534\.7±\\pm0\.237\.8±\\pm1\.333\.3±\\pm2\.532\.6±\\pm2\.834\.6Qwen2\.5\-7B\-Instruct \(Strong\)Llama3\-8B\-Instruct \(Strong\)MethodAlpacaEval 2Arena\-HardAvg\.AlpacaEval 2Arena\-HardAvg\.LCWRSCWRLCWRSCWRBase32\.3±\\pm0\.430\.2±\\pm1\.538\.3±\\pm2\.240\.1±\\pm2\.835\.228\.1±\\pm0\.328\.1±\\pm1\.324\.7±\\pm2\.525\.2±\\pm2\.726\.5Weak SFT \(pre\)17\.8±\\pm0\.317\.2±\\pm1\.427\.4±\\pm2\.631\.3±\\pm2\.723\.413\.7±\\pm0\.414\.8±\\pm1\.614\.1±\\pm2\.813\.8±\\pm2\.514\.1Weak SFT \(post\)33\.0±\\pm0\.331\.0±\\pm1\.338\.9±\\pm2\.440\.7±\\pm2\.735\.928\.7±\\pm0\.328\.8±\\pm1\.525\.3±\\pm2\.625\.9±\\pm2\.527\.2WeakTeacher36\.8±\\pm0\.438\.7±\\pm1\.646\.5±\\pm2\.552\.0±\\pm2\.843\.530\.8±\\pm0\.432\.1±\\pm1\.633\.5±\\pm2\.734\.0±\\pm2\.832\.6AuxConf21\.2±\\pm0\.220\.7±\\pm1\.327\.2±\\pm2\.432\.1±\\pm2\.625\.316\.7±\\pm0\.519\.5±\\pm1\.714\.3±\\pm2\.313\.5±\\pm2\.616\.0WSPO18\.6±\\pm0\.321\.0±\\pm1\.229\.1±\\pm2\.233\.7±\\pm2\.925\.617\.3±\\pm0\.419\.8±\\pm1\.618\.5±\\pm2\.116\.9±\\pm2\.418\.1ConG\-S\(w→\\rightarrows\)38\.7±\\pm0\.543\.1±\\pm1\.752\.4±\\pm2\.557\.8±\\pm2\.948\.033\.7±\\pm0\.234\.3±\\pm1\.439\.6±\\pm2\.339\.2±\\pm2\.836\.7ConG\(w→\\rightarrows\)43\.0±\\pm0\.451\.9±\\pm1\.654\.5±\\pm2\.661\.2±\\pm2\.752\.738\.3±\\pm0\.341\.2±\\pm1\.543\.6±\\pm2\.243\.3±\\pm2\.941\.6

In this section, we conduct extensive experiments to address the following research questions:

- •RQ1:How does our proposed ConG\-S and ConG perform compared to baseline approaches in both weak\-to\-strong alignment and self\-alignment settings?
- •RQ2:How does the effectiveness of ConG\-S and ConG in weak\-to\-strong alignment vary with different capability gaps between the weak and strong models?
- •RQ3:How does the contrastive coefficientα\\alphainfluence the performance of ConG\-S and ConG?
- •RQ4:How do ConG\-S and ConG affect downstream evaluations, and to what extent do they preserve the model’s general capabilities without degradation?

### 5\.1Experimental Setup

In this subsection, we summarize the base models, dataset, benchmarks and baselines used in our experiments\. Further details and additional experiments are provided in Appendix[B](https://arxiv.org/html/2510.07884#A2)and[D](https://arxiv.org/html/2510.07884#A4)\.

Base Models and Dataset\.We conduct experiments on two widely used model families: Qwen2\.5\(Yanget al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib25)\)and Llama3\(Dubeyet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib24)\)\. For weak\-to\-strong alignment, we follow prior works\(Zhuet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib18); Lyuet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib31)\)and select the smaller\-parameter models within each family, Qwen2\.5\-3B\-Instruct and Llama3\.2\-3B\-Instruct, as the reference weak modelsπrefweak\\pi\_\{\\mathrm\{ref\}\}^\{\\mathrm\{weak\}\}, and the larger\-parameter counterparts, Qwen2\.5\-7B\-Instruct and Llama3\-8B\-Instruct, as the reference strong modelsπrefstrong\\pi\_\{\\mathrm\{ref\}\}^\{\\mathrm\{strong\}\}\. We further construct the aligned weak modelπrweak\\pi\_\{r\}^\{\\mathrm\{weak\}\}by fine\-tuning each reference weak model with DPO, which is then used to facilitate contrastive decoding and provide chosen samples\. For preference alignment data, we adopt the UltraFeedback dataset\(Cuiet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib30)\)\. To construct pairwise supervision for DPO training, to rank the self\-generated samples from the reference weak models and generate pairwise annotations\.

Evaluation Benchmarks and Baselines\.We evaluate our method on two widely used instruction\-following benchmarks, AlpacaEval2\(Duboiset al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib22)\)and Arena\-Hard\(Liet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib23)\)\. For AlpacaEval2, we report both the length\-controlled win rate \(LC\) and the raw win rate \(WR\)\. For Arena\-Hard, we report the win rate \(WR\) and the style\-controlled win rate \(SC\)\. For baselines, we adopt different experimental settings for weak and strong models\. For weak models, we evaluate our method in the self\-alignment setting \(ConG\-S\(self\)and ConG\(self\)\) against preference optimization algorithms, including DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2510.07884#bib.bib10)\), ORPO\(Honget al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib32)\), and SimPO\(Menget al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib33)\)\. For strong models, we focus on the weak\-to\-strong alignment setting, where supervision signals must come from the weak model for fairness\. We therefore evaluate our method \(ConG\-S\(w→\\rightarrows\)and ConG\(w→\\rightarrows\)\) against established weak\-to\-strong approaches: Weak SFT \(pre\)\(Burnset al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib11)\), which uses the pre\-alignment weak model; Weak SFT \(post\), which instead uses the post\-alignment weak model; WeakTeacher\(Tao and Li,[2025](https://arxiv.org/html/2510.07884#bib.bib54)\); AuxConf\(Burnset al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib11)\), a variant of Weak SFT with an auxiliary confidence loss; and WSPO\(Zhuet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib18)\)\.

![Refer to caption](https://arxiv.org/html/2510.07884v2/figures/exp_2.png)Figure 3:Performance of contrastive weak\-to\-strong generalization\. \(a\) and \(b\) Results across different weak–strong model\. \(c\) and \(d\) Effect of contrastive coefficientα\\alphaon alignment performance, with error bars indicating standard errors\. “Base” refers to the unaligned reference model\.
### 5\.2Weak\-to\-Strong Alignment Results \(RQ1\)

We first evaluate the effectiveness of our method in both the self\-alignment and weak\-to\-strong alignment settings\. Specifically, we use Qwen2\.5\-3B\-Instruct and Llama3\.2\-3B\-Instruct as weak models to improve the performance of the Qwen2\.5\-7B\-Instruct and Llama3\-8B\-Instruct strong models, respectively\. The results are summarized in Table[1](https://arxiv.org/html/2510.07884#S5.T1), from which we find that:

- •Obs 1: In the self\-alignment setting, our method significantly improves over existing preference optimization baselines\.Specifically, ConG\-S\(self\)and ConG\(self\)achieve average improvements of about15\.0%and17\.8%, respectively, compared to reference\. This highlights that contrastive decoding outputs from models indeed provide higher\-quality supervision signals\.
- •Obs 2: In the weak\-to\-strong setting, our method achieves substantial improvements over the base models\.Specifically, ConG\-S\(w→\\rightarrows\)and ConG\(w→\\rightarrows\)improve the average scores by around11\.5%and16\.3%, respectively\. These results demonstrate that contrastive decoding effectively transfers preference information from weak to strong models, leading to stronger alignment\. Notably, other baselines even degrade performance relative to the base models\.

### 5\.3Effect of Model Capability Gap \(RQ2\)

From the results in Table[1](https://arxiv.org/html/2510.07884#S5.T1), we observe that our method yields larger gains in self\-alignment than in weak\-to\-strong alignment\. This raises the question of whether the capability gap between weak and strong models affects the improvement\. Based on the experiments across different model transitions, the results in Figure[3](https://arxiv.org/html/2510.07884#S5.F3)\(a\) and \(b\) lead to the following observations:

- •Obs 3: Smaller model capability gaps lead to larger improvements in weak\-to\-strong alignment\.Specifically, transitions where the weak and strong models have closer sizes \(e\.g\.,7B→7B\) result in significantly higher alignment improvements compared to those with larger capability gaps \(e\.g\.,1\.5B→7B\)\. This indicates that when the weak model is closer in capacity to the strong model, the weak model can provide more effective supervision, leading to better alignment\.
- •Obs 4: Larger strong models contribute to more substantial weak\-to\-strong alignment improvements\.Specifically, transitions involving stronger strong models \(e\.g\.,3B→7B\) show significantly larger performance gains compared to transitions with smaller strong models \(e\.g\.,1\.5B→3B\)\. This suggests that larger strong models are better at leveraging the preference signals from weak models, resulting in stronger alignment when trained on these signals\.

### 5\.4Impact ofα\\alphaon Weak\-to\-Strong Alignment \(RQ3\)

We study the effect of the contrastive coefficientα\\alphaon weak\-to\-strong alignment performance\. As shown in Figure[3](https://arxiv.org/html/2510.07884#S5.F3)\(c\) and \(d\), we report results across differentα\\alphavalues\. The figures show that:

- •Obs 5: Moderate values ofα\\alpharesult in the best weak\-to\-strong alignment performance\.We find that settingα\\alphato moderate values \(e\.g\.,α=0\.3\\alpha=0\.3toα=0\.5\\alpha=0\.5\) consistently yields the highest performance gains\. These values allow the contrastive term to significantly enhance the weak model’s preference signal without overwhelming the original model behavior\.
- •Obs 6: Whenα\>0\.5\\alpha\>0\.5, performance improvement diminishes with increasingα\\alpha\.Afterα\\alphaexceeds 0\.5, further increases inα\\alphalead to diminishing returns in alignment performance\. Specifically, asα\\alphagrows larger, the additional benefits from the contrastive signal become less significant, and the overall improvement plateaus\.

Table 2:Downstream task results on Qwen2\.5\-7B\-Instruct and Llama3\-8B\-Instruct\.Differences relative to the Base are shown in parentheses\. Improvements are marked inred, decreases inblue\.MethodARC\_EARC\_CTruthfulQAMMLUMathQAHellaSwagGSM8KQwen2\.5\-7B\-InstructBase81\.31 \(\+0\.00\)52\.82 \(\+0\.00\)50\.18 \(\+0\.00\)71\.80 \(\+0\.00\)40\.74 \(\+0\.00\)62\.04 \(\+0\.00\)80\.44 \(\+0\.00\)Weak SFT76\.02 \(\-5\.29\)50\.01 \(\-2\.81\)47\.12 \(\-3\.06\)67\.55 \(\-4\.25\)38\.12 \(\-2\.62\)61\.44 \(\-0\.60\)75\.28 \(\-5\.16\)AuxConf77\.81 \(\-3\.50\)51\.37 \(\-1\.45\)49\.24 \(\-0\.94\)68\.12 \(\-3\.68\)41\.02 \(\+0\.28\)60\.85 \(\-1\.19\)76\.42 \(\-4\.02\)WSPO78\.25 \(\-3\.06\)49\.88 \(\-2\.94\)48\.71 \(\-1\.47\)70\.22 \(\-1\.58\)39\.11 \(\-1\.63\)62\.55 \(\+0\.51\)77\.08 \(\-3\.36\)ConG\-S80\.44 \(\-0\.87\)52\.21 \(\-0\.61\)50\.75 \(\+0\.57\)72\.12 \(\+0\.32\)39\.95 \(\-0\.79\)61\.55 \(\-0\.49\)79\.12 \(\-1\.32\)ConG81\.02 \(\-0\.29\)52\.64 \(\-0\.18\)50\.28 \(\+0\.10\)71\.42 \(\-0\.38\)40\.51 \(\-0\.23\)62\.88 \(\+0\.84\)80\.21 \(\-0\.23\)Llama3\-8B\-InstructBase81\.40 \(\+0\.00\)52\.99 \(\+0\.00\)46\.76 \(\+0\.00\)63\.81 \(\+0\.00\)42\.01 \(\+0\.00\)57\.71 \(\+0\.00\)75\.28 \(\+0\.00\)Weak SFT75\.83 \(\-5\.57\)50\.45 \(\-2\.54\)44\.28 \(\-2\.48\)59\.14 \(\-4\.67\)39\.22 \(\-2\.79\)54\.03 \(\-3\.68\)71\.12 \(\-4\.16\)AuxConf77\.42 \(\-3\.98\)51\.82 \(\-1\.17\)45\.89 \(\-0\.87\)60\.92 \(\-2\.89\)41\.44 \(\-0\.57\)55\.18 \(\-2\.53\)72\.08 \(\-3\.20\)WSPO78\.04 \(\-3\.36\)49\.91 \(\-3\.08\)46\.42 \(\-0\.34\)62\.08 \(\-1\.73\)40\.36 \(\-1\.65\)58\.02 \(\+0\.31\)73\.21 \(\-2\.07\)ConG\-S80\.62 \(\-0\.78\)52\.05 \(\-0\.94\)47\.19 \(\+0\.43\)64\.01 \(\+0\.20\)41\.12 \(\-0\.89\)57\.25 \(\-0\.46\)74\.38 \(\-0\.90\)ConG81\.22 \(\-0\.18\)52\.71 \(\-0\.28\)46\.93 \(\+0\.17\)63\.54 \(\-0\.27\)42\.41 \(\+0\.40\)57\.38 \(\-0\.33\)75\.66 \(\+0\.38\)

### 5\.5Downstream Task Evaluation \(RQ4\)

To assess whether ConG\-based weak\-to\-strong generalization affects downstream task performance, we evaluate models trained with different alignment methods on a diverse suite of benchmarks from the lm\-eval\-harness\(Sutawikaet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib46)\)\. The evaluation covers MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2510.07884#bib.bib47)\), ARC\(Clarket al\.,[2018](https://arxiv.org/html/2510.07884#bib.bib48)\), HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2510.07884#bib.bib49)\), TruthfulQA\(Linet al\.,[2022](https://arxiv.org/html/2510.07884#bib.bib50)\), MathQA\(Aminiet al\.,[2019](https://arxiv.org/html/2510.07884#bib.bib52)\), and GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2510.07884#bib.bib51)\)\. We follow standard evaluation protocols and report the results in Table[2](https://arxiv.org/html/2510.07884#S5.T2), enabling a direct comparison of whether ConG preserves general capabilities\. Based on these results, we draw the following observation:

- •Obs 7: ConG introduces negligible degradation on downstream tasks\.We observe that ConG and ConG\-S maintain general\-purpose capabilities nearly intact, with average changes within1\.01\.0points across benchmarks, indicating that our method preserves overall model utility\.

## 6Related Work

AI alignment\.A central challenge in modern AI research is ensuring that advanced models act in line with human intent\(Caoet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib14); Gaoet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib15); Leikeet al\.,[2018](https://arxiv.org/html/2510.07884#bib.bib1); Askellet al\.,[2021](https://arxiv.org/html/2510.07884#bib.bib2)\)\. Current alignment pipelines often depend on human feedback, most prominently RLHF\(Ouyanget al\.,[2022](https://arxiv.org/html/2510.07884#bib.bib9); Christianoet al\.,[2017](https://arxiv.org/html/2510.07884#bib.bib8); Stiennonet al\.,[2020](https://arxiv.org/html/2510.07884#bib.bib7)\)and DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2510.07884#bib.bib10)\)\. Recent studies have further extended alignment considerations to broader deployment settings, including safety\-aware training and intervention mechanisms\(Fanget al\.,[2026a](https://arxiv.org/html/2510.07884#bib.bib59); Jianget al\.,[2026b](https://arxiv.org/html/2510.07884#bib.bib63)\)\. While effective at present, such approaches still scale poorly: human supervision becomes insufficient once models surpass expert ability, and collecting reliable annotations remains expensive and difficult\(Kimet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib41); Zenget al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib42)\)\. This challenge is also increasingly relevant beyond purely text\-based models, as recent work explores alignment and reasoning in multimodal systems\(Jianget al\.,[2026a](https://arxiv.org/html/2510.07884#bib.bib62)\)\. These limitations motivate alignment strategies that reduce or bypass reliance on direct human oversight\.

Weak\-to\-strong generalization\.An alternative line of work explores whether stronger models can be trained under the supervision of weaker ones, a setting known as weak\-to\-strong generalization \(W2SG\)\. Initial evidence by\(Burnset al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib11)\)showed that strong models may outperform their weak teachers, pointing to the potential of this paradigm\. Subsequent studies introduced new algorithms\(Zhuet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib18); Lyuet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib31); Liet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib34); Somerstepet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib35); Zhouet al\.,[2024b](https://arxiv.org/html/2510.07884#bib.bib28); Mitchellet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib21); Zhouet al\.,[2024a](https://arxiv.org/html/2510.07884#bib.bib20); Tao and Li,[2025](https://arxiv.org/html/2510.07884#bib.bib54)\)and provided empirical analyses across tasks\(Yaoet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib16); Yanget al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib38)\)\. Relatedly, distillation\-based approaches also aim to transfer stronger behavioral signals into smaller or specialized models, including recent on\-policy variants for language models and agents\(Zhonget al\.,[2026](https://arxiv.org/html/2510.07884#bib.bib60); Fanget al\.,[2026b](https://arxiv.org/html/2510.07884#bib.bib61)\)\. Compared with these efforts, our work highlights a different perspective: we connect W2SG with implicit rewards and contrastive decoding, offering both conceptual justification and empirical validation on large language models\.

Contrastive Decoding\.Contrastive decoding lies at the core of our work\. The key idea is to refine generation by contrasting probability distributions during decoding\. The original contrastive decoding method\(Liet al\.,[2023](https://arxiv.org/html/2510.07884#bib.bib17)\)contrasts a stronger expert model with a weaker amateur model to improve fluency and coherence\. Subsequent extensions explore different contrastive dimensions: DoLa\(Chuanget al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib44)\)contrasts later, more mature layers with earlier layers to stabilize generation, while ICD\(Zhanget al\.,[2023](https://arxiv.org/html/2510.07884#bib.bib45)\)contrasts models perturbed with hallucination\-inducing noise to enhance factual accuracy\. These studies highlight the flexibility of contrastive decoding and motivate our use of it as the foundation for W2SG\.

## 7Limitations and Future Works

Our work has several limitations\. First, contrastive decoding is not yet fully compatible with mainstream inference acceleration techniques\. Even though we adopt a pre\-generation design to mitigate overhead, the additional computation still introduces latency compared to standard decoding, which may limit efficiency in practice\. Second, ConG relies on maintaining multiple alignment states of the weak model to enable contrastive decoding\. While the added complexity is moderate, it nevertheless increases the engineering burden relative to simpler pipelines\.

To address these issues, we see several promising directions\. A natural next step is to explore adapting contrastive decoding to fast inference paradigms such as speculative decoding and caching\-based acceleration, which could greatly reduce latency\. In addition, designing lighter\-weight strategies to approximate multiple weak\-model alignment states may simplify deployment without sacrificing effectiveness\. We also plan to evaluate ConG in larger\-scale, real\-world applications, providing deeper insights into its practicality and robustness\.

## 8Conclusion

We presentedContrastive Weak\-to\-Strong Generalization \(ConG\), a new paradigm that connects implicit rewards with contrastive decoding\. Our key insight is that implicit rewards, parameterized as log\-likelihood ratios, are structurally consistent with the mechanism of contrastive decoding\. This correlation allows us to view contrastive decoding not only as a decoding strategy but also as a natural way of generating samples that maximize implicit reward\. Building on this connection, ConG leverages contrastive decoding between aligned model states to provide higher\-quality supervision signals, enabling more effective capability transfer, denoising, and improved robustness\.

## Impact Statement

This work studies weak\-to\-strong generalization for LLMs and proposes a contrastive\-decoding framework that reduces noise and bias in weak\-model\-generated training data without human feedback or explicit reward modeling, aiming to improve robustness and lower the cost of scalable alignment\. The approach may benefit low\-resource and domain\-adaptation settings and help mitigate error amplification in training pipelines, but it can still inherit and propagate biases from the weak models if used without careful evaluation\. We therefore emphasize responsible deployment, including dataset curation, testing across diverse settings, and complementary safety mechanisms\.

## Acknowledgment

This research/project was supported by National Natural Science Foundation of China \(U25A20445\), the National Natural Science Foundation of China\(62525211\), the Zhongguancun Academy, and the Singapore Ministry of Education \(MOE\) Academic Research Fund \(AcRF\) Tier 1 grant \(Proposal ID: 24\-SIS\-SMU\-002\)\. Yang Deng is support by the Lee Kong Chian Fellowship awarded by Singapore Management University\.

## References

- A\. Amini, S\. Gabriel, P\. Lin, R\. Koncel\-Kedziorski, Y\. Choi, and H\. Hajishirzi \(2019\)Mathqa: towards interpretable math word problem solving with operation\-based formalisms\.arXiv preprint arXiv:1905\.13319\.Cited by:[§5\.5](https://arxiv.org/html/2510.07884#S5.SS5.p1.1)\.
- A\. Askell, Y\. Bai, A\. Chen, D\. Drain, D\. Ganguli, T\. Henighan, A\. Jones, N\. Joseph, B\. Mann, N\. DasSarma, N\. Elhage, Z\. Hatfield\-Dodds, D\. Hernandez, J\. Kernion, K\. Ndousse, C\. Olsson, D\. Amodei, T\. B\. Brown, J\. Clark, S\. McCandlish, C\. Olah, and J\. Kaplan \(2021\)A general language assistant as a laboratory for alignment\.CoRRabs/2112\.00861\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- R\. A\. Bradley and M\. E\. Terry \(1952\)Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§2\.1](https://arxiv.org/html/2510.07884#S2.SS1.p1.8)\.
- C\. Burns, P\. Izmailov, J\. H\. Kirchner, B\. Baker, L\. Gao, L\. Aschenbrenner, Y\. Chen, A\. Ecoffet, M\. Joglekar, J\. Leike, I\. Sutskever, and J\. Wu \(2024\)Weak\-to\-strong generalization: eliciting strong capabilities with weak supervision\.InICML,Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p1.1),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p3.2),[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- B\. Cao, K\. Lu, X\. Lu, J\. Chen, M\. Ren, H\. Xiang, P\. Liu, Y\. Lu, B\. He, X\. Han, L\. Sun, H\. Lin, and B\. Yu \(2024\)Towards scalable automated alignment of llms: A survey\.CoRRabs/2406\.01252\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. B\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.InNIPS,pp\. 4299–4307\.Cited by:[§2\.1](https://arxiv.org/html/2510.07884#S2.SS1.p1.12),[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- Y\. Chuang, Y\. Xie, H\. Luo, Y\. Kim, J\. R\. Glass, and P\. He \(2024\)DoLa: decoding by contrasting layers improves factuality in large language models\.InICLR,Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p3.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the AI2 reasoning challenge\.CoRRabs/1803\.05457\.Cited by:[§5\.5](https://arxiv.org/html/2510.07884#S5.SS5.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.CoRRabs/2110\.14168\.Cited by:[§5\.5](https://arxiv.org/html/2510.07884#S5.SS5.p1.1)\.
- G\. Cui, L\. Yuan, N\. Ding, G\. Yao, B\. He, W\. Zhu, Y\. Ni, G\. Xie, R\. Xie, Y\. Lin, Z\. Liu, and M\. Sun \(2024\)ULTRAFEEDBACK: boosting language models with scaled AI feedback\.InICML,Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p5.1),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p2.3)\.
- G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, W\. Li, B\. He, Y\. Fan, T\. Yu, Q\. Xu, W\. Chen, J\. Yuan, H\. Chen, K\. Zhang, X\. Lv, S\. Wang, Y\. Yao, X\. Han, H\. Peng, Y\. Cheng, Z\. Liu, M\. Sun, B\. Zhou, and N\. Ding \(2025\)Process reinforcement through implicit rewards\.CoRRabs/2502\.01456\.Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p3.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Rozière, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. M\. Kloumann, I\. Misra, I\. Evtimov, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, and et al\. \(2024\)The llama 3 herd of models\.CoRRabs/2407\.21783\.Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p5.1),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p2.3)\.
- Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto \(2024\)Length\-controlled alpacaeval: A simple way to debias automatic evaluators\.CoRRabs/2404\.04475\.Cited by:[§B\.1](https://arxiv.org/html/2510.07884#A2.SS1.p1.1),[§1](https://arxiv.org/html/2510.07884#S1.p5.1),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p3.2)\.
- J\. Fang, N\. Chen, H\. Jiang, D\. Zhang, F\. Shen, X\. Wang, X\. He, and T\. Chua \(2026a\)NExT\-guard: training\-free streaming safeguard without token\-level labels\.CoRRabs/2603\.02219\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- J\. Fang, Z\. Hong, M\. Zheng, M\. Song, G\. Li, H\. Jiang, D\. Zhang, H\. Guo, X\. Wang, and T\. Chua \(2026b\)Rubric\-based on\-policy distillation\.arXiv preprint arXiv:2605\.07396\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- B\. Gao, F\. Song, Y\. Miao, Z\. Cai, Z\. Yang, L\. Chen, H\. Hu, R\. Xu, Q\. Dong, C\. Zheng, W\. Xiao, G\. Zhang, D\. Zan, K\. Lu, B\. Yu, D\. Liu, Z\. Cui, J\. Yang, L\. Sha, H\. Wang, Z\. Sui, P\. Wang, T\. Liu, and B\. Chang \(2024\)Towards a unified view of preference learning for large language models: A survey\.CoRRabs/2409\.02795\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- B\. Goertzel \(2014\)Artificial general intelligence: concept, state of the art, and future prospects\.Journal of Artificial General Intelligence5\(1\),pp\. 1\.Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InICLR,Cited by:[§5\.5](https://arxiv.org/html/2510.07884#S5.SS5.p1.1)\.
- J\. Hong, N\. Lee, and J\. Thorne \(2024\)ORPO: monolithic preference optimization without reference model\.InEMNLP,pp\. 11170–11189\.Cited by:[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p3.2)\.
- H\. Jiang, J\. Fu, J\. Fang, C\. Gao, X\. Wang, X\. He, and Y\. Li \(2026a\)UniVLR: unifying text and vision in visual latent reasoning for multimodal llms\.arXiv preprint arXiv:2605\.11856\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- H\. Jiang, Z\. Zhao, J\. Fang, H\. Ma, R\. Wang, X\. Wang, X\. He, and Y\. Deng \(2026b\)DualEdit: mitigating safety fallback in llm backdoor editing via affirmation\-refusal regulation\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- H\. Kim, X\. Yi, J\. Yao, J\. Lian, M\. Huang, S\. Duan, J\. Bak, and X\. Xie \(2024\)The road to artificial superintelligence: A comprehensive survey of superalignment\.CoRRabs/2412\.16468\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- H\. Lee, S\. Phatale, H\. Mansoor, T\. Mesnard, J\. Ferret, K\. Lu, C\. Bishop, E\. Hall, V\. Carbune, A\. Rastogi, and S\. Prakash \(2024\)RLAIF vs\. RLHF: scaling reinforcement learning from human feedback with AI feedback\.InICML,Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p1.1)\.
- J\. Leike, D\. Krueger, T\. Everitt, M\. Martic, V\. Maini, and S\. Legg \(2018\)Scalable agent alignment via reward modeling: a research direction\.CoRRabs/1811\.07871\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- T\. Li, W\. Chiang, E\. Frick, L\. Dunlap, T\. Wu, B\. Zhu, J\. E\. Gonzalez, and I\. Stoica \(2024\)From crowdsourced data to high\-quality benchmarks: arena\-hard and benchbuilder pipeline\.CoRRabs/2406\.11939\.Cited by:[§B\.1](https://arxiv.org/html/2510.07884#A2.SS1.p1.1),[§1](https://arxiv.org/html/2510.07884#S1.p5.1),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p3.2)\.
- X\. L\. Li, A\. Holtzman, D\. Fried, P\. Liang, J\. Eisner, T\. Hashimoto, L\. Zettlemoyer, and M\. Lewis \(2023\)Contrastive decoding: open\-ended text generation as optimization\.InACL \(1\),pp\. 12286–12312\.Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p3.1),[§3\.2](https://arxiv.org/html/2510.07884#S3.SS2.p1.4),[§6](https://arxiv.org/html/2510.07884#S6.p3.1)\.
- Y\. Li, X\. Miao, M\. Xu, and T\. Qian \(2025\)Strong empowered and aligned weak mastered annotation for weak\-to\-strong generalization\.InAAAI,pp\. 27437–27445\.Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p1.1),[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)TruthfulQA: measuring how models mimic human falsehoods\.InACL \(1\),pp\. 3214–3252\.Cited by:[§5\.5](https://arxiv.org/html/2510.07884#S5.SS5.p1.1)\.
- Y\. Lyu, L\. Yan, Z\. Wang, D\. Yin, P\. Ren, M\. de Rijke, and Z\. Ren \(2025\)MACPO: weak\-to\-strong alignment via multi\-agent contrastive preference optimization\.InICLR,Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p2.1),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p2.3),[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- Y\. Meng, M\. Xia, and D\. Chen \(2024\)SimPO: simple preference optimization with a reference\-free reward\.InNeurIPS,Cited by:[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p3.2)\.
- E\. Mitchell, R\. Rafailov, A\. Sharma, C\. Finn, and C\. D\. Manning \(2024\)An emulator for fine\-tuning large language models using small language models\.InICLR,Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- OpenAI \(2023\)GPT\-4 technical report\.CoRRabs/2303\.08774\.Cited by:[§B\.1](https://arxiv.org/html/2510.07884#A2.SS1.p2.1),[§B\.1](https://arxiv.org/html/2510.07884#A2.SS1.p3.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p1.1),[§2\.1](https://arxiv.org/html/2510.07884#S2.SS1.p1.12),[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p3.1),[§2\.2](https://arxiv.org/html/2510.07884#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p3.2),[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.CoRRabs/1707\.06347\.Cited by:[§2\.1](https://arxiv.org/html/2510.07884#S2.SS1.p1.12)\.
- S\. Somerstep, F\. M\. Polo, M\. Banerjee, Y\. Ritov, M\. Yurochkin, and Y\. Sun \(2025\)A transfer learning framework for weak to strong generalization\.InICLR,Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p1.1),[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano \(2020\)Learning to summarize with human feedback\.InNeurIPS,Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- L\. Sutawika, H\. Schoelkopf, L\. Gao, B\. Abbasi, S\. Biderman, J\. Tow, ben fattori, C\. Lovering, farzanehnakhaee70, J\. Phang, A\. Thite, Fazz, Aflah, N\. Muennighoff, T\. Wang, sdtblck, nopperl, gakada, tttyuntian, researcher2, J\. Etxaniz, Chris, H\. A\. Lee, Z\. Kasner, Khalid, LSinev, J\. Hsu, A\. Kanekar, KonradSzafer, and AndyZwei \(2024\)EleutherAI/lm\-evaluation\-harness: v0\.4\.3External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://doi.org/10.5281/zenodo.12608602)Cited by:[§5\.5](https://arxiv.org/html/2510.07884#S5.SS5.p1.1)\.
- L\. Tao and Y\. Li \(2025\)Your weak LLM is secretly a strong teacher for alignment\.InICLR,Cited by:[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p3.2),[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- H\. Wang, W\. Xiong, T\. Xie, H\. Zhao, and T\. Zhang \(2024\)Interpretable preferences via multi\-objective reward modeling and mixture\-of\-experts\.InEMNLP \(Findings\),pp\. 10582–10592\.Cited by:[§B\.2](https://arxiv.org/html/2510.07884#A2.SS2.p3.4)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2024\)Qwen2\.5 technical report\.CoRRabs/2412\.15115\.Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p5.1),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p2.3)\.
- W\. Yang, S\. Shen, G\. Shen, W\. Yao, Y\. Liu, G\. Zhi, Y\. Lin, and J\. Wen \(2025\)Super\(ficial\)\-alignment: strong models may deceive weak models in weak\-to\-strong generalization\.InICLR,Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p2.1),[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- W\. Yao, W\. Yang, Z\. Wang, Y\. Lin, and Y\. Liu \(2025\)Understanding the capabilities and limitations of weak\-to\-strong generalization\.CoRRabs/2502\.01458\.Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p1.1),[§1](https://arxiv.org/html/2510.07884#S1.p2.1),[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- L\. Yuan, W\. Li, H\. Chen, G\. Cui, N\. Ding, K\. Zhang, B\. Zhou, Z\. Liu, and H\. Peng \(2024\)Free process rewards without process labels\.CoRRabs/2412\.01981\.Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p3.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InACL \(1\),pp\. 4791–4800\.Cited by:[§5\.5](https://arxiv.org/html/2510.07884#S5.SS5.p1.1)\.
- Y\. Zeng, F\. Zhao, Y\. Wang, E\. Lu, Y\. Yang, L\. Wang, C\. Liu, Y\. Liang, D\. Zhao, B\. Han,et al\.\(2025\)Redefining superalignment: from weak\-to\-strong alignment to human\-ai co\-alignment to sustainable symbiotic society\.arXiv preprint arXiv:2504\.17404\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p1.1)\.
- Y\. Zhang, L\. Cui, W\. Bi, and S\. Shi \(2023\)Alleviating hallucinations of large language models through induced hallucinations\.arXiv preprint arXiv:2312\.15710\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p3.1)\.
- Q\. Zhong, M\. Zheng, M\. Song, X\. Lin, J\. Sun, H\. Jiang, X\. Wang, and J\. Fang \(2026\)SOD: step\-wise on\-policy distillation for small language model agents\.arXiv preprint arXiv:2605\.07725\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- Z\. Zhou, J\. Liu, Z\. Dong, J\. Liu, C\. Yang, W\. Ouyang, and Y\. Qiao \(2024a\)Emulated disalignment: safety alignment for large language models may backfire\!\.InACL \(1\),pp\. 15810–15830\.Cited by:[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- Z\. Zhou, Z\. Liu, J\. Liu, Z\. Dong, C\. Yang, and Y\. Qiao \(2024b\)Weak\-to\-strong search: align large language models via searching over small language models\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2510.07884#S1.p1.1),[§3\.1](https://arxiv.org/html/2510.07884#S3.SS1.p1.3),[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.
- W\. Zhu, Z\. He, X\. Wang, P\. Liu, and R\. Wang \(2025\)Weak\-to\-strong preference optimization: stealing reward from weak aligned model\.InICLR,Cited by:[§B\.2](https://arxiv.org/html/2510.07884#A2.SS2.p5.1.1),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p2.3),[§5\.1](https://arxiv.org/html/2510.07884#S5.SS1.p3.2),[§6](https://arxiv.org/html/2510.07884#S6.p2.1)\.

## Appendix AAlgorithmic Details of ConG

We provide the unified algorithm for Contrastive Weak\-to\-Strong Generalization \(ConG\), which consists of two stages: \(i\) ConG\-S, using contrastive decoding \(CD\) responses for SFT, and \(ii\) ConG, refining the strong model with DPO\.

Algorithm 1Contrastive Weak\-to\-Strong Generalization \(ConG\)Input:Pre\-alignment weak modelπrefw\\pi^\{\\mathrm\{w\}\}\_\{\\mathrm\{ref\}\}, post\-alignment weak modelπrw\\pi^\{\\mathrm\{w\}\}\_\{r\}, initial strong modelπrefs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{ref\}\}, prompt set𝒳\\mathcal\{X\}, contrastive coefficientα\\alpha, preference sharpnessβ\\beta\.

Output:Aligned strong modelπs\\pi^\{\\mathrm\{s\}\}\.

Stage I: ConG\-S \(Contrastive Decoding for SFT\) for*each promptx∈𝒳x\\in\\mathcal\{X\}*do

Generate CD response

ywy\_\{w\}from

\(πrw,πrefw,α\)\(\\pi^\{\\mathrm\{w\}\}\_\{r\},\\pi^\{\\mathrm\{w\}\}\_\{\\mathrm\{ref\}\},\\alpha\)\. Add

\(x,yw\)\(x,y\_\{w\}\)into dataset

𝒟SFT\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}\.

Train

πrefs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{ref\}\}on

𝒟SFT\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}with SFT loss \(Eqn\.[10](https://arxiv.org/html/2510.07884#S4.E10)\) to obtain

πSFTs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}\.Stage II: ConG \(Generalization with DPO\) for*each promptx∈𝒳x\\in\\mathcal\{X\}*do

Retrieve

ywy\_\{w\}from

𝒟SFT\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}\. Sample additional response

yl∼πSFTsy\_\{l\}\\sim\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}with standard decoding\. Construct pair

\(yw,yl\)\(y\_\{w\},y\_\{l\}\)with

rw\>rlr\_\{w\}\>r\_\{l\}\. Add

\(x,yw,yl\)\(x,y\_\{w\},y\_\{l\}\)into dataset

𝒟DPO\\mathcal\{D\}\_\{\\mathrm\{DPO\}\}\.

Train

πSFTs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}on

𝒟DPO\\mathcal\{D\}\_\{\\mathrm\{DPO\}\}with DPO loss \(Eqn\.[11](https://arxiv.org/html/2510.07884#S4.E11)\) to obtain final strong model

πs\\pi^\{\\mathrm\{s\}\}\.

## Appendix BExperimental Setup

### B\.1Benchmarks

Our evaluation is conducted on two widely adopted benchmarks for instruction\-following language models:AlpacaEval2\(Duboiset al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib22)\)andArena\-Hard\(Liet al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib23)\)\. Both benchmarks follow a pairwise comparison paradigm, in which the output of a tested model is directly compared against that of a reference model on a shared prompt, and a stronger external judge model decides which response better satisfies the instruction\. This setup provides a scalable and reliable approximation of human preference judgments, while controlling for common biases such as verbosity and stylistic artifacts\. Below we provide detailed descriptions of these two benchmarks, the evaluation methodology, and the specific settings used in our experiments\.

AlpacaEval2\.AlpacaEval2 is the successor of AlpacaEval, designed to address issues of bias and instability in preference\-based evaluation\. It consists of around 805 diverse prompts that span open\-ended question answering, reasoning, creative writing, and general instruction\-following tasks\. The evaluation protocol compares model responses head\-to\-head with those of a reference baseline model, with each pair adjudicated by a strong LLM judge\. The main metric is the raw*win rate \(WR\)*, defined as the proportion of cases where the evaluated model’s response is preferred over the baseline’s\. To reduce artifacts, AlpacaEval2 additionally reports the*length\-controlled win rate \(LC\)*: a variant that normalizes for verbosity bias, since longer responses often receive higher preference regardless of actual quality\. Length control is performed by conditioning on response length differences and recalibrating the win rate, thereby offering a fairer view of content quality\. Beyond length, the benchmark also considers stylistic features such as formatting or lexical variety, and the*style\-controlled win rate*removes such confounds by balancing stylistic attributes across comparisons\. These refinements make AlpacaEval2 one of the most widely used and trusted benchmarks for instruction tuning\. In our experiments, we follow the standard practice of using GPT\-4\-1106\-preview\(OpenAI,[2023](https://arxiv.org/html/2510.07884#bib.bib43)\)both as the reference model and as the judge\. The reference ensures a consistent baseline, while GPT\-4 as the judge provides stable and high\-quality preference assessments\.

Arena\-Hard\.Arena\-Hard was specifically developed to capture more challenging evaluation scenarios\. Derived from the Chatbot Arena data and further curated with automatic and manual filtering, Arena\-Hard consists of difficult, real\-world style prompts designed to separate strong models from one another\. Compared with AlpacaEval2, Arena\-Hard emphasizes separability and agreement with human preferences\. Separability means that the benchmark is capable of distinguishing models with small quality differences, reflected in tighter confidence intervals and less overlap in rankings\. Human agreement means that model rankings generated by Arena\-Hard correlate strongly with actual human preference data, an essential property for trustworthy evaluation\. Like AlpacaEval2, Arena\-Hard uses pairwise head\-to\-head comparisons, reporting raw win rates and, where possible,*style\-controlled win rates*to mitigate superficial biases such as excessive formatting, verbosity, or stylistic quirks\. This makes Arena\-Hard particularly suitable for evaluating strong models where subtle quality differences matter\. In our setup, we adopt GPT\-4–0314\(OpenAI,[2023](https://arxiv.org/html/2510.07884#bib.bib43)\)as the reference model against which all tested models are compared, while GPT\-4\-1106\-preview is used as the judge model to evaluate the outputs\. This combination has become a de facto standard in recent works because it balances fairness, stability, and robustness of evaluation\.

### B\.2Implementation Details

Contrastive Decoding\.For all experiments involving contrastive decoding \(CD\), we adopt the vocabulary pruning mechanism proposed in the original CD paper, with pruning thresholdλ∈\[0,1\]\\lambda\\in\[0,1\]fixed at0\.10\.1across all settings\. This pruning step restricts the candidate vocabulary at each decoding step and effectively reduces spurious low\-probability tokens\. The contrastive coefficientα\\alphais tuned separately for different weak models via grid search: for Qwen2\.5\-3B\-Instruct we selectα=0\.5\\alpha=0\.5, and for Llama3\.2\-3B\-Instruct we selectα=0\.4\\alpha=0\.4\. These values achieve a balance between reward maximization and distributional stability\. To mitigate repetition artifacts in generations, we follow the original CD implementation and set the repetition penalty coefficient to1\.21\.2\. For the sampling strategy, we use greedy decoding, as our focus is on extracting responses that best approximate implicit reward maximization rather than increasing output diversity\.

SFT Training\.For the supervised fine\-tuning \(SFT\) stage, we adopt the LLaMA\-Factory framework with DeepSpeed ZeRO\-3 optimization\. Unless otherwise stated, all hyperparameters are consistent across models\. The learning rate is set to1\.0×10−51\.0\\times 10^\{\-5\}, chosen from a search range of\{5\.0×10−6,1\.0×10−5,2\.0×10−5\}\\\{5\.0\\times 10^\{\-6\},1\.0\\times 10^\{\-5\},2\.0\\times 10^\{\-5\}\\\}\. Training is run for22epochs with a cosine learning rate scheduler and a warmup ratio of0\.10\.1\. The maximum response length is fixed at20482048tokens\. These settings provide a stable training regime while avoiding overfitting, and they are consistent with widely adopted practices for weak\-to\-strong alignment\.

DPO Training\.For the DPO stage, we again use the LLaMA\-Factory framework with DeepSpeed ZeRO\-3\. For standard DPO, we use a learning rate of8\.0×10−78\.0\\times 10^\{\-7\}and setβ=0\.1\\beta=0\.1, following prior work\. The preference dataset for DPO is constructed via a standard pipeline: the LLM generates five candidate responses for each prompt, which are then scored by the reward modelArmoRM\-Llama3\-8B\-v0\.1\(Wanget al\.,[2024](https://arxiv.org/html/2510.07884#bib.bib29)\), with the highest\-scoring response taken asywy\_\{w\}and the lowest\-scoring asyly\_\{l\}\.

In contrast, our ConG method adopts a slightly different configuration: we set the learning rate to6\.0×10−76\.0\\times 10^\{\-7\}, chosen from a search range of4\.0×10−7,6\.0×10−7,8\.0×10−7\{4\.0\\times 10^\{\-7\},6\.0\\times 10^\{\-7\},8\.0\\times 10^\{\-7\}\}, and useβ=0\.5\\beta=0\.5\. The higherβ\\betavalue reflects the fact that\(yw,yl\)\(y\_\{w\},y\_\{l\}\)pairs are only approximately drawn from the same distribution—depending on the fidelity of ConG\-S—so a stronger preference scaling helps prevent the optimization from drifting too far from the implicit reward signal\.

WSPO\(Zhuet al\.,[2025](https://arxiv.org/html/2510.07884#bib.bib18)\)\.Weak\-to\-Strong Preference Optimization \(WSPO\) extends the idea of weak\-to\-strong generalization to alignment settings\. The method leverages the observation that the alignment behavior learned by a weaker model can be transferred—and even amplified—by a stronger model\. Concretely, WSPO optimizes the strong model to capture the distributional shift exhibited by the weak model before and after its own alignment stage\. In the original formulation, WSPO relies on an externally annotated response dataset to supervise this preference shift\. However, our experimental setup focuses strictly on self\-generated weak\-model responses, without using any external aligned outputs\. To ensure a fair comparison, we therefore replace WSPO’s external response data with responses sampled from the weak model itself\. This modification naturally leads to a reduction in performance, but it better matches the weak\-to\-strong generalization setting, where the goal is to improve the strong model by exploiting the informative structure present in weak\-model generations rather than introducing additional supervised signals\.

Baselines\.We also re\-implement several preference optimization baselines for comparison\. For ORPO, we searchλ∈\{0\.1,0\.5,1\.0,2\.0\}\\lambda\\in\\\{0\.1,0\.5,1\.0,2\.0\\\}as suggested in the original paper\. For SimPO, we tune overβ∈\{2\.0,4\.0,6\.0,8\.0\}\\beta\\in\\\{2\.0,4\.0,6\.0,8\.0\\\}andγ∈\{0\.3,0\.5,1\.0,1\.2,1\.4,1\.6\}\\gamma\\in\\\{0\.3,0\.5,1\.0,1\.2,1\.4,1\.6\\\}\. For WSPO, we follow the default settings in the original work, with learning rate1\.0×10−51\.0\\times 10^\{\-5\},β=0\.1\\beta=0\.1, and training for11epoch\. For all baselines, the maximum response length is consistently set to20482048tokens to ensure comparability with our method\.

Computation Environment\.All training experiments are conducted using 4×NVIDIA L40 GPUs, with mixed precision training enabled\. Our experimental pipeline follows the official guidelines provided in the LLaMA\-Factory repository, ensuring reproducibility and alignment with established community practices\. We also employ gradient checkpointing and ZeRO\-3 optimizer states to maximize memory efficiency\.

## Appendix CSupplementary Proofs

In this section, we provide theoretical support for the role of contrastive decoding \(CD\) in maximizing implicit reward and justify the preference structure used in ConG\. We focus on three aspects: \(i\) the correlation between CD and implicit reward maximization, \(ii\) the implicit reward gap between CD and naive sampling, and \(iii\) the relative reward ordering between CD responses and ConG\-S generations\.

#### CD–Implicit Reward Correlation\.

Recall the token\-level implicit reward:

r^t​\(x,y<t,yt\)=log⁡πr​\(yt∣x,y<t\)πref​\(yt∣x,y<t\)\.\\hat\{r\}\_\{t\}\(x,y\_\{<t\},y\_\{t\}\)=\\log\\frac\{\\pi\_\{r\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\}\.At decoding steptt, CD defines the sampling distribution as

pα​\(yt∣x,y<t\)∝exp⁡\(\(1−α\)​r^t​\(x,y<t,yt\)\+α​log⁡πr​\(yt∣x,y<t\)\),α∈\[0,1\]\.p\_\{\\alpha\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\\propto\\exp\\\!\\left\(\(1\-\\alpha\)\\,\\hat\{r\}\_\{t\}\(x,y\_\{<t\},y\_\{t\}\)\+\\alpha\\log\\pi\_\{r\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\\right\),\\quad\\alpha\\in\[0,1\]\.\(12\)This can be rewritten as

pα​\(yt\)∝πr​\(yt\)​exp⁡\(\(1−α\)​r^t​\(yt\)\),p\_\{\\alpha\}\(y\_\{t\}\)\\propto\\pi\_\{r\}\(y\_\{t\}\)\\exp\\\!\\big\(\(1\-\\alpha\)\\hat\{r\}\_\{t\}\(y\_\{t\}\)\\big\),showing that CD is precisely an exponential tilting ofπr\\pi\_\{r\}by the statisticr^t\\hat\{r\}\_\{t\}\. Hence, CD samples tokens that maximize a weighted combination of implicit reward andπr\\pi\_\{r\}’s likelihood, and in the limitα→0\\alpha\\\!\\to\\\!0, CD reduces to pure implicit\-reward maximization\.

#### Reward Gap between CD and Naive Sampling\.

LetyCDy^\{\\mathrm\{CD\}\}denote a token sampled frompαp\_\{\\alpha\}, andynaivey^\{\\mathrm\{naive\}\}fromπr\\pi\_\{r\}\(the caseα=1\\alpha=1\)\. The expected implicit reward underpαp\_\{\\alpha\}satisfies

𝔼pα​\[r^t\]≥𝔼πr​\[r^t\],\\mathbb\{E\}\_\{p\_\{\\alpha\}\}\[\\hat\{r\}\_\{t\}\]\\;\\geq\\;\\mathbb\{E\}\_\{\\pi\_\{r\}\}\[\\hat\{r\}\_\{t\}\],with equality only whenα=1\\alpha=1\. To see this, defineZ​\(η\)=∑vπr​\(v\)​exp⁡\(η​r^t​\(v\)\)Z\(\\eta\)=\\sum\_\{v\}\\pi\_\{r\}\(v\)\\exp\(\\eta\\hat\{r\}\_\{t\}\(v\)\), whereη=1−α\\eta=1\-\\alpha\. Then

𝔼pα​\[r^t\]=∂∂η​log⁡Z​\(η\)\.\\mathbb\{E\}\_\{p\_\{\\alpha\}\}\[\\hat\{r\}\_\{t\}\]=\\frac\{\\partial\}\{\\partial\\eta\}\\log Z\(\\eta\)\.Since∂η2log⁡Z​\(η\)=Varpα​\[r^t\]≥0\\partial^\{2\}\_\{\\eta\}\\log Z\(\\eta\)=\\mathrm\{Var\}\_\{p\_\{\\alpha\}\}\[\\hat\{r\}\_\{t\}\]\\geq 0, the function is non\-decreasing inη\\eta\. Thus, moving away fromα=1\\alpha=1\(i\.e\., increasing the contrastive weight\) strictly increases the expected implicit reward\. Summing overttextends the result to full responses, establishing that CD consistently yields higher implicit reward than naive sampling\.

#### Reward Ordering between CD and ConG\-S Generations\.

In Stage I \(ConG\-S\), we train the strong modelπSFTs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}on responsesywy\_\{w\}drawn from the weak\-model CD distributionpαwp\_\{\\alpha\}^\{\\mathrm\{w\}\}\. Formally,

pαw​\(yt\)∝πrw​\(yt\)​exp⁡\(\(1−α\)​r^tw\)\.p\_\{\\alpha\}^\{\\mathrm\{w\}\}\(y\_\{t\}\)\\propto\\pi\_\{r\}^\{\\mathrm\{w\}\}\(y\_\{t\}\)\\exp\(\(1\-\\alpha\)\\hat\{r\}\_\{t\}^\{\\mathrm\{w\}\}\)\.The SFT procedure minimizesDKL​\(pαw∥πθs\)D\_\{\\mathrm\{KL\}\}\(p\_\{\\alpha\}^\{\\mathrm\{w\}\}\\parallel\\pi^\{\\mathrm\{s\}\}\_\{\\theta\}\), projectingpαwp\_\{\\alpha\}^\{\\mathrm\{w\}\}onto the strong model family\. UnlessπSFTs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}can perfectly representpαwp\_\{\\alpha\}^\{\\mathrm\{w\}\}, the projection attenuates the tilting effect, leading to

𝔼πSFTs​\[r^w\]≤𝔼pαw​\[r^w\],\\mathbb\{E\}\_\{\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}\}\[\\hat\{r\}^\{\\mathrm\{w\}\}\]\\;\\leq\\;\\mathbb\{E\}\_\{p\_\{\\alpha\}^\{\\mathrm\{w\}\}\}\[\\hat\{r\}^\{\\mathrm\{w\}\}\],and therefore

𝔼​\[r^​\(x,yl\)\]≤𝔼​\[r^​\(x,yw\)\],\\mathbb\{E\}\[\\hat\{r\}\(x,y\_\{l\}\)\]\\;\\leq\\;\\mathbb\{E\}\[\\hat\{r\}\(x,y\_\{w\}\)\],whereywy\_\{w\}is a CD response andyly\_\{l\}is a sample fromπSFTs\\pi^\{\\mathrm\{s\}\}\_\{\\mathrm\{SFT\}\}\.

This establishes the preference orderingyw≻yly\_\{w\}\\succ y\_\{l\}used in Stage II \(DPO\), providing the theoretical justification for ConG’s design\.

#### Summary\.

Together, these results show that: \(i\) CD is an implicit\-reward–maximizing decoder, \(ii\) CD responses have strictly higher expected implicit reward than naive sampling, and \(iii\) ConG\-S generations cannot exceed the implicit reward of their CD teacher, which justifies pairing\(yw,yl\)\(y\_\{w\},y\_\{l\}\)as preference data in Stage II\.

## Appendix DAdditional Experiments

### D\.1Cross\-Family Weak\-to\-Strong Alignment

To further examine the generality of our proposed ConG framework, we extend the evaluation to a more challenging*cross\-family weak\-to\-strong setting*, where the weak and strong models come from different model families\. Specifically, we consider two scenarios: \(i\) aligningLlama3\.2\-3B\-Instructas the weak model to guideQwen2\.5\-7B\-Instructas the strong model, and \(ii\) aligningQwen2\.5\-3B\-Instructas the weak model to guideLlama3\-8B\-Instructas the strong model\. We use the same UltraFeedback dataset and training protocol as in the in\-family experiments, ensuring fairness and comparability\. Baselines include Weak SFT, AuxConf, WSPO, and standard preference optimization methods\.

#### Results\.

Table[3](https://arxiv.org/html/2510.07884#A4.T3)reports the results on AlpacaEval2 and Arena\-Hard\. We observe that cross\-family alignment remains highly effective under ConG, though performance is slightly lower compared to in\-family alignment\. In the Llama3\.2\-3B→\\rightarrowQwen2\.5\-7B setting, ConG achieves an average score of 51\.7, significantly outperforming all baselines\. In contrast, the Qwen2\.5\-3B→\\rightarrowLlama3\-8B setting shows even stronger improvements, where ConG reaches an average score of 42\.1, surpassing alternative methods by a large margin\. These results suggest that ConG not only generalizes across scales within the same model family, but also transfers effectively across heterogeneous architectures\.

#### Observation\.

The overall trend indicates that cross\-family alignment is feasible and beneficial: while absolute performance is slightly lower than in\-family settings \(with a gap of roughly1\.01\.0–1\.51\.5points on average\), ConG still delivers consistent gains over strong baselines\. This demonstrates that contrastive decoding provides a robust preference signal that transcends model families, further validating the broad applicability of the proposed framework\.

Table 3:Cross\-family weak\-to\-strong results on AlpacaEval2 and Arena\-Hard\.Top: Llama3\.2\-3B→\\rightarrowQwen2\.5\-7B\. Bottom: Qwen2\.5\-3B→\\rightarrowLlama3\-8B\. “WR” denotes the raw win rate, “LC” the length\-controlled win rate, “SC” the style\-controlled win rate, and “Avg\.” the average across benchmarks\. Best results are in bold and second\-best are underlined\. Numbers are mean±\\pmstd\.Llama3\.2\-3B→\\rightarrowQwen2\.5\-7BQwen2\.5\-3B→\\rightarrowLlama3\-8BMethodAlpacaEval 2Arena\-HardAvg\.AlpacaEval 2Arena\-HardAvg\.LCWRSCWRLCWRSCWRBase32\.3±\\pm0\.430\.2±\\pm1\.538\.3±\\pm2\.240\.1±\\pm2\.835\.228\.1±\\pm0\.328\.1±\\pm1\.324\.7±\\pm2\.525\.2±\\pm2\.726\.5Weak SFT18\.1±\\pm0\.317\.5±\\pm1\.427\.1±\\pm2\.531\.0±\\pm2\.723\.414\.1±\\pm0\.415\.0±\\pm1\.614\.5±\\pm2\.814\.2±\\pm2\.514\.4AuxConf21\.0±\\pm0\.220\.3±\\pm1\.327\.0±\\pm2\.431\.6±\\pm2\.625\.017\.0±\\pm0\.519\.9±\\pm1\.714\.8±\\pm2\.313\.9±\\pm2\.616\.4WSPO18\.4±\\pm0\.320\.7±\\pm1\.228\.7±\\pm2\.233\.2±\\pm2\.925\.317\.6±\\pm0\.420\.2±\\pm1\.618\.9±\\pm2\.117\.4±\\pm2\.418\.5ConG\-S\(w→\\rightarrows\)37\.9±\\pm0\.542\.2±\\pm1\.751\.5±\\pm2\.656\.6±\\pm2\.947\.134\.6±\\pm0\.235\.1±\\pm1\.440\.7±\\pm2\.340\.5±\\pm2\.837\.7ConG\(w→\\rightarrows\)42\.1±\\pm0\.450\.6±\\pm1\.653\.8±\\pm2\.560\.4±\\pm2\.751\.739\.2±\\pm0\.342\.5±\\pm1\.544\.9±\\pm2\.244\.7±\\pm2\.942\.8

### D\.2Training Consumption

To evaluate the computational efficiency of our approach, we report the training time \(GPU hours on 4×L40\) for all weak\-to\-strong baselines and our methods\. As shown in Table[4](https://arxiv.org/html/2510.07884#A4.T4), ConG\-S and ConG exhibit comparable training costs to standard weak\-to\-strong approaches such as Weak SFT and WSPO, with no additional overhead introduced during optimization\. This is expected because the core contribution of our method lies in its offline data construction process: the contrastive guidance signals are generated through a lightweight sampling procedure that can be fully performed before training begins\.

Thus, unlike methods that modify the training loop or introduce additional forward passes, our approach does not require extra compute during fine\-tuning\. All models are trained under the same regime, and the empirical GPU\-hour measurements confirm that ConG\-S and ConG maintain essentially the same training cost as existing baselines, while achieving consistently stronger performance\.

Table 4:Training time \(GPU hours\) on 4×L40\.MethodQwen2\.5\-7BLlama3\-8BWeak SFT \(pre\)10\.6811\.42Weak SFT \(post\)11\.1811\.78WSPO16\.9718\.38ConG\-S10\.7811\.48ConG13\.8014\.78
### D\.3Cross Judging

To further validate the robustness of our method under different evaluation protocols, we additionally conduct experiments using two complementary judges: \(i\) GPT\-5, a stronger but more conservative automatic evaluator, and \(ii\) a small portion \(10%\) of human raters\. As automatic judges are known to vary in strictness across model families, these evaluations help assess whether our improvements persist under different scoring behaviors\.

As shown in Table[5](https://arxiv.org/html/2510.07884#A4.T5)and[6](https://arxiv.org/html/2510.07884#A4.T6), across both Qwen2\.5\-7B and Llama3\-8B, we observe that GPT\-5 tends to assign slightly lower absolute scores, while human evaluators generally give slightly higher ones\. However, the relative ordering of all methods remains unchanged, demonstrating that the performance gains brought by ConG and ConG\-S are not artifacts of a specific judge\. Importantly, under all judges—including GPT\-4, GPT\-5, and human raters—ConG and ConG\-S consistently achieve the best results, while baseline weak\-to\-strong methods exhibit the same relative ranking\.

These findings confirm that our approach provides stable and judge\-invariant alignment improvements, and further reinforce that the benefits of contrastive weak\-to\-strong alignment hold across different evaluation standards, model families, and scoring paradigms\.

Table 5:Results on AlpacaEval2 and Arena\-Hard under GPT\-5 Judger\.Qwen2\.5\-7B\-InstructLlama3\-8B\-InstructMethodAlpacaEval 2Arena\-HardAvg\.AlpacaEval 2Arena\-HardAvg\.LCWRSCWRLCWRSCWRBase27\.025\.133\.034\.930\.023\.022\.419\.320\.121\.2Weak SFT \(pre\)12\.611\.922\.125\.318\.09\.09\.69\.08\.89\.1Weak SFT \(post\)28\.025\.834\.035\.531\.023\.623\.320\.121\.022\.0WeakTeacher31\.132\.039\.343\.636\.525\.626\.728\.028\.727\.3AuxConf16\.716\.022\.927\.020\.712\.014\.010\.49\.711\.5WSPO14\.816\.924\.928\.121\.212\.814\.513\.412\.513\.3ConG\-S33\.837\.947\.052\.042\.728\.729\.634\.934\.832\.0ConG37\.646\.449\.355\.847\.333\.336\.038\.738\.536\.6

Table 6:Results on AlpacaEval2 and Arena\-Hard with 10% Human Evaluation\.Qwen2\.5\-7B\-InstructLlama3\-8B\-InstructMethodAlpacaEval 2Arena\-HardAvg\.AlpacaEval 2Arena\-HardAvg\.LCWRSCWRLCWRSCWRBase35\.533\.441\.343\.138\.331\.130\.927\.428\.329\.4Weak SFT \(pre\)20\.920\.131\.034\.326\.617\.017\.817\.216\.617\.2Weak SFT \(post\)36\.734\.642\.043\.839\.231\.832\.028\.729\.730\.6WeakTeacher40\.041\.149\.051\.545\.433\.734\.936\.336\.935\.4AuxConf24\.624\.031\.535\.829\.019\.521\.916\.715\.918\.5WSPO23\.024\.232\.936\.329\.118\.820\.619\.418\.319\.3ConG\-S42\.246\.355\.260\.451\.036\.437\.243\.043\.240\.0ConG45\.854\.257\.563\.255\.240\.643\.146\.546\.244\.1

### D\.4Case Study

Case Study on Llama3\-8B\-InstructInstructionWho wears short shorts?BASEThe iconic question\!Who wears short shorts? Well, it’s Kevin Naismith, of course\!ConG\-SThe term ”shorts” can refer to various individuals from different professions or groups who wear them as part of their uniform or for casual wear\. Here are a few examples:1\. Athletes: Many sports players, such as baseball players, soccer players, and basketball players, wear shorts as part of their uniforms\.2\. Runners and joggers: People who engage in running or jogging as exercise often wear shorts for comfort and practicality\.3\. Skaters: Skaters, including ice skaters \(ice hockey, figure skating\) and inline skaters, wear specialized shorts designed for their respective sports\.4\. Cyclists: Some cyclists, especially those who wear lycra or stretchy clothing, may wear shorts for comfort and mobility\.5\. Pool\-goers: Lifeguards, swim instructors, and beachgoers often wear shorts for practicality and sun protection\.6\. Golfers: While not exclusively, some golfers wear shorts in warmer weather or as a fashion choice\.7\. Shorts\-wearing celebrities: You might spot famous individuals like Drake, David Beckham \(former soccer player turned fashion icon\), and other stars who often incorporate shorts into their everyday or red\-carpet outfits\.These are just a few examples, but the list goes on, as shorts are a versatile and popular clothing item enjoyed by people of all ages and backgrounds\!Who do you have in mind when asking ”Who wears short shorts?”ConGThe phrase ”short shorts” can be interpreted in a few ways:1\. Athletes and sports players often wear short shorts as part of their uniforms, particularly in warm weather or for specific sports like basketball, soccer, and gymnastics\.2\. In casual, summery settings, some people might wear short shorts as a fashion choice or for comfort\.3\. In the context of ”who wears short shorts,” the answer could be anyone who chooses to don this style, regardless of age, gender, or activity level\!To give a more specific answer, I’ll say that people from various walks of life can be found wearing short shorts, but it’s more commonly associated with:\- Younger generations \(kids, teenagers, and young adults\) \- Athletes and sports enthusiasts \- People living in warm climates or during hot weather months \- Those who prioritize comfort and practicality over traditional dress codesLet me know if you’d like more clarification\!

Case Study on Qwen2\.5\-7B\-InstructInstructionWhat is kevlar made out of?BASEKevlar is a synthetic fiber known for its high strength\-to\-weight ratio and heat resistance\.It is primarily made from a polyamide polymer, which is derived from a combination of terephthalic acid and hexamethylenediamine\.The specific chemical structure and manufacturing process give Kevlar its remarkable properties, making it widely used in various applications such as body armor, protective clothing, tires, and composite materials for sports equipment and construction\.ConG\-SKevlar is primarily made from synthetic polymers, specifically aromatic polyamides\. More precisely:\- Kevlar is synthesized from a monomer called terephthalic acid \(TA\) and a diamine called 1,4\-phenylene diamine \(PDA\)\. \- The process involves chemical reactions to form polymer chains, which are then spun into fibers\.Key points: 1\.Monomers: The basic building blocks are polymers derived from terephthalic acid and PDA\. 2\.Polymer Structure: The resulting polymer forms strong, flexible molecules with high tensile strength\. 3\.Manufacturing: These polymers are processed into fibers through methods like melt spinning or wet spinning\.Thus, while not a natural material, Kevlar’s core composition and properties are defined by these synthetic polymer structures\.It’s widely used for its exceptional strength\-to\-weight ratio, making it ideal for applications like body armor, bulletproof vests, and protective gear in various industries\.ConGKevlar is primarily made from synthetic polymers, specifically poly\(p\-phenylene terephthalamide\)\. Here’s a detailed breakdown:1\.Chemical Composition: Kevlar is composed of aromatic polyamides\. Its basic structure involves long chains of repeating units derived from terephthalic acid and para\-phenylene diamine\.2\.Manufacturing Process: \-Raw Materials: The primary raw materials are terephthalic acid and p\-phenylene diamine \(PPD\)\. \-Polymerization: These monomers undergo polymerization through a process called polycondensation, typically under controlled conditions, to form the long polymer chains that make up Kevlar\. \-Drawing and Orientation: After polymerization, the resulting fibers are drawn and oriented to increase their strength and reduce their density, making them stronger yet lighter\.3\.Properties Derived: The specific arrangement and chemical bonds in these polymeric chains give Kevlar its exceptional strength\-to\-weight ratio, heat resistance, and excellent tensile properties\.In essence, while not a simple elemental composition but rather a complex polymer, Kevlar’s fundamental building blocks come from the aforementioned chemical precursors\.This material has wide\-ranging applications due to its unique properties, from reinforcing tires and bulletproof vests to composite materials in various industries\.