How Does Post-Training Shape LLM Value Preferences?

# How Does Post-Training ShapeLLM Value Preferences?
Source: [https://arxiv.org/html/2610.33371](https://arxiv.org/html/2610.33371)
Shaoyang XuAffiliation:Singapore University of Technology and DesignEmail:[chu@i\.kyoto\-u\.ac\.jp](mailto:)Masahiro KanekoAffiliation:Mohamed bin Zayed University of Artificial IntelligenceChengzhi ZhongAffiliation:Kyoto UniversityWenxuan ZhangAffiliation:Singapore University of Technology and DesignChenhui ChuAffiliation:Kyoto University

###### Abstract

During post\-training, large language models \(LLMs\) learn value preferences from corpora that may contain imbalanced proportions of opinions associated with different demographic groups\. Amplifying these imbalances may further underrepresent some groups’ opinions\. However, how opinion mixtures shape value preferences across post\-training paths remains unclear\. To answer this question, we conduct controlled experiments using post\-training corpora with different opinion mixtures and five post\-training paths based on SFT, DPO, and GRPO\. We find that post\-training amplifies opinion imbalances differently across post\-training paths\. Once value preferences are amplified, whether further post\-training can remove them depends on both the post\-training method and the opinion mixture in the corpus\. More concerningly, a model’s stability under interventions depends on whether further post\-training changes its value preference\. Moreover, these value preferences are transferred to student models even when distillation uses corpora with no explicit opinion questions\. These findings suggest that both the opinion mixture in post\-training corpora and the choice of post\-training method should be considered when developing, evaluating, and reusing LLMs\. Our code and data are available at[https://github\.com/ku\-nlp/llm\-value\-preferences](https://github.com/ku-nlp/llm-value-preferences)\.

## 1Introduction

Large language model \(LLM\) post\-training commonly uses supervised learning from demonstrations, preference optimization from comparisons, and reinforcement learning from reward signals\([Ouyang et al\., 2022](https://arxiv.org/html/2610.33371#bib.bib39);[Rafailov et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib44);[Shao et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib51)\)\. Although these methods use different learning signals, each relies on a post\-training corpus that can contain opinions associated with different demographic groups\([Kirk et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib22);[Sap et al\., 2022](https://arxiv.org/html/2610.33371#bib.bib49)\)\. Even when developers do not intentionally favor any group, these opinions may be unevenly represented in the corpus\. During post\-training, these corpus imbalances may shape the model’s value preferences\([Chen et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib3)\)and be amplified in its responses, causing the model to underrepresent some groups’ opinions\.

Existing studies show that imbalanced training corpora shape models’ biases and value preferences\. For example, gender imbalances in training corpora can shape the gender bias of masked language models\([Kaneko et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib21)\)\. Visual recognition models can further amplify gender imbalances in their training datasets\([Zhao et al\., 2017](https://arxiv.org/html/2610.33371#bib.bib70);[Hendricks et al\., 2018](https://arxiv.org/html/2610.33371#bib.bib14)\)\. The composition of post\-training corpora can also shape the values expressed by LLMs during supervised fine\-tuning \(SFT\)\([Chen et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib3)\)\. When groups hold conflicting preferences, preference optimization over\-represents the majority preference\([Yao et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib68);[Xiao et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib64)\)\. Moreover, even with the same post\-training corpus, different post\-training methods can produce different value preferences\([Bhatia et al\., 2026](https://arxiv.org/html/2610.33371#bib.bib1)\)\.

However, how post\-training methods and the opinion mixture in the corpus jointly shape LLM value preferences has not been systematically compared\. In particular, it remains unclear how the order of post\-training stages shapes value preferences and how further post\-training can remove the learned value preferences\. It is also unknown how a model’s post\-training history affects both the stability of its learned preferences under interventions such as prompt injection\([Perez & Ribeiro, 2022](https://arxiv.org/html/2610.33371#bib.bib41)\)and their transfer to other models through distillation\([Hinton et al\., 2015](https://arxiv.org/html/2610.33371#bib.bib16)\)\.

To address these gaps, we conduct controlled mixture experiments based on the OpinionQA dataset\([Santurkar et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib48)\), which contains survey questions and response distributions across demographic groups\. Specifically, we construct post\-training corpora whose examples match the majority answer of one of two demographic groups\. We construct 11 post\-training corpora with one group’s opinion represented from 0% to 100% in 10% increments\. The remaining examples reflect the other group’s opinion\. We then apply SFT, direct preference optimization\([Rafailov et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib44), DPO;\), and group relative policy optimization\([Shao et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib51), GRPO;\)\. These methods define five post\-training paths: SFT alone, or SFT followed by DPO, GRPO, or both in different orders\. Finally, each model is evaluated separately on training\-covered and training\-uncovered questions\.

We find that all five post\-training paths amplify the corpus opinion imbalance on the training\-covered set\. By contrast, on the training\-uncovered set, LLM value preferences change with the mixture proportion between 0\.3 and 0\.7 but saturate beyond this range\. When an LLM already holds a value preference, further SFT using a balanced corpus can restore equal alignment, but other methods cannot guarantee equal alignment with the same corpus\. For example, removing a model’s existing value preference for one group with DPO can require the post\-training corpus to contain more opinions from the opposing group than from the preferred group\.

We also examine the stability of learned value preferences under interventions and their transfer through model reuse\. Their stability under interventions depends on the relationship between a model’s first\-stage and final\-stage preferences\. For example, models whose first\-stage preference is neutral or is reversed by later stages show higher flip rates\. When LLMs with learned value preferences are reused as teachers, task\-only distillation transfers these preferences to student models, creating a risk of unintended preference propagation\.

Our findings highlight the need to evaluate LLM value preferences throughout model development and reuse\. This is particularly important for widely used open\-weight models, which are further post\-trained through multi\-stage pipelines\([Lambert et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib25);[Grattafiori et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib11);[Walsh et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib58);[Yang et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib67)\)and distilled into smaller models of other families\([Guo et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib12)\)\.

## 2Experimental Setup

### 2\.1Opinion Post\-Training Corpus

![Refer to caption](https://arxiv.org/html/2610.33371v1/example.png)Figure 1:An OpinionQA question and each group’s majority answer\.To build the opinion post\-training corpus, we use OpinionQA\([Santurkar et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib48)\),111OpinionQA is derived from the Pew Research Center’s American Trends Panel \(ATP\)\. The url of OpinionQA is[https://github\.com/tatsu\-lab/opinions\_qa](https://github.com/tatsu-lab/opinions_qa)which contains multiple\-choice survey questions about opinions, with each question having 2 to 5 options\. Figure[1](https://arxiv.org/html/2610.33371#S2.F1)illustrates an example from OpinionQA\.

##### Demographic Attributes and Groups\.

A demographic attribute is a characteristic used to group respondents and a group consists of respondents who share a particular value of an attribute\([Santurkar et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib48)\)\. We consider three attributes:party,education, andreligion\. We compare two groups for each attribute, as listed in Table[6](https://arxiv.org/html/2610.33371#A2.T6)in Appendix[B\.1](https://arxiv.org/html/2610.33371#A2.SS1)\.

##### Questions and Answers\.

We use the group’s majority answer to represent that group’s opinion on a given question\.222Appendix[B\.1](https://arxiv.org/html/2610.33371#A2.SS1)shows the answer distributions of the two groups on each question for the party attribute\.Although the majority answer of a group does not capture differences of opinion among group members\([Sorensen et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib53);[Liu et al\., 2026a](https://arxiv.org/html/2610.33371#bib.bib34)\), it provides a reproducible way to measure whether an LLM’s answers are closer to one group’s majority answers than to another’s\. We retain only questions for which the two groups have different answers, which leaves 200 questions on party, 156 on education, and 104 on religion\.

##### Post\-Training Corpus Augmentation\.

We have limited data for post\-training, so we need to expand the corpus\. Following the semantic data augmentation \(SDA\) method of[Li et al\. \(2024a\)](https://arxiv.org/html/2610.33371#bib.bib28), we expand each question into 25 semantically equivalent variants\. For each attribute, we divide questions into two sets\. We perform SDA on only one of the sets, and we only use the augmented data for post\-training; the original questions are not used in the post\-training process\. Appendix[B\.1](https://arxiv.org/html/2610.33371#A2.SS1)provides details and specific examples of SDA\.

### 2\.2Mixture Proportion

As shown in the example in Figure[1](https://arxiv.org/html/2610.33371#S2.F1), if the answer following a question reflects the opinion of a group, we consider that data point to represent that group’s opinion\. For each attribute, we denote the two groups byg1g\_\{1\}andg2g\_\{2\}\(group 1 and group 2 in Table[6](https://arxiv.org/html/2610.33371#A2.T6)\), and we use the mixture proportionppto denote the proportion of the corpus that reflectsg1g\_\{1\}’s opinion\. We construct corpora for each of the eleven proportionsp∈\{0\.0,0\.1,…,1\.0\}p\\in\\\{0\.0,0\.1,\\dots,1\.0\\\}\. For example, ifp=0\.3p=0\.3, then 30% of the examples in the corpus reflectg1g\_\{1\}’s opinion, and the remaining 70% reflectg2g\_\{2\}’s opinion\.

### 2\.3Models

Our main experiments post\-train three LLMs:Llama\-3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib11)\),Qwen3\-8B\-Base\([Yang et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib67)\), andOLMo\-2\-7B\([Walsh et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib58)\)\. All three are base models, allowing us to study the effects of our controlled post\-training corpora without prior instruction tuning\. In the distillation experiments, the Llama\-3\.1\-8B checkpoints serve as teacher models, while Qwen3\-1\.7B\-Base, Llama\-3\.2\-3B, and OLMo\-2\-1B serve as student models \(Section[6](https://arxiv.org/html/2610.33371#S6)\)\. For the robustness analyses, we use Qwen3\-4B\-Base to compare low\-rank adaptation\([Hu et al\., 2021](https://arxiv.org/html/2610.33371#bib.bib17), LoRA;\)with full\-parameter fine\-tuning \(Appendix[G\.4](https://arxiv.org/html/2610.33371#A7.SS4)\) and Qwen3\-8B, Llama\-3\.1\-8B\-Instruct, and OLMo\-2\-7B\-Instruct to examine the effect of the starting model \(Appendix[G\.5](https://arxiv.org/html/2610.33371#A7.SS5)\)\.

### 2\.4Post\-Training

We post\-train each LLM in three stages: SFT, DPO, and GRPO\. The three stages form a pipeline, so each stage starts from the LLM that the previous stage produced\. We always use SFT as the first stage of post\-training\. This is because SFT teaches the LLM to provide an option number at the answer position, which is a behavior required for all subsequent stages as well as for evaluation\. We give the objectives in Appendix[B\.5](https://arxiv.org/html/2610.33371#A2.SS5)and the hyperparameters in Appendix[A\.3](https://arxiv.org/html/2610.33371#A1.SS3)\.

Because each stage starts from the checkpoint produced by the preceding stage, a single sequence cannot distinguish the effect of the current stage from the effects inherited from earlier stages\. We therefore compare five post\-training paths: SFT alone \(SFT\); SFT followed by DPO \(S2D\); SFT followed by GRPO \(S2G\); SFT followed by DPO and then GRPO \(SDG\); and SFT followed by GRPO and then DPO \(SGD\)\. Unless stated otherwise, we use a balanced corpus for every stage preceding the final stage\. The starting checkpoints for the final post\-training stage have agreement scores close to 0\.5 in most cases and therefore show no strong preference for either group \(Appendix[B\.6](https://arxiv.org/html/2610.33371#A2.SS6)\)\.

### 2\.5Evaluation Question Sets

In standard task evaluation, overlap between training and evaluation questions should be avoided because it can cause data leakage\([Lewis et al\., 2021](https://arxiv.org/html/2610.33371#bib.bib27);[Magar & Schwartz, 2022](https://arxiv.org/html/2610.33371#bib.bib36)\)\. Opinion alignment has a different practical objective: developers may seek to cover the opinion questions they can identify, although the wording in the post\-training corpus may differ from the wording encountered after training\([Li et al\., 2024a](https://arxiv.org/html/2610.33371#bib.bib28);[Li et al\., 2024b](https://arxiv.org/html/2610.33371#bib.bib29)\)\. We therefore evaluate two sets separately to examine how the LLM’s value preferences change under different levels of corpus coverage\.

For each attribute, we use the following two sets:

Training\-Covered Set: It consists of questions used for SDA\. Therefore, the semantics of this set of questions are covered by the training corpus\.

Training\-Uncovered Set: It consists of questions not used for SDA\. Therefore, the semantics of this set of questions are not covered by the training corpus\.

### 2\.6Metrics

For a demographic attribute with groupsg1g\_\{1\}andg2g\_\{2\}, letℳp\\mathcal\{M\}\_\{p\}denote the LLM post\-trained at mixture proportionpp\. To characterize the LLM’s value preferences, we define the following two scores\.

##### Agreement Score\.

Letng1​\(p\)n\_\{g\_\{1\}\}\(p\)andng2​\(p\)n\_\{g\_\{2\}\}\(p\)be the numbers of questions for whichℳp\\mathcal\{M\}\_\{p\}’s answer matches the majority answer ofg1g\_\{1\}andg2g\_\{2\}, respectively\.333Appendix[C\.4](https://arxiv.org/html/2610.33371#A3.SS4)compares this score with a distribution\-based score\.ng1​\(p\)n\_\{g\_\{1\}\}\(p\)andng2​\(p\)n\_\{g\_\{2\}\}\(p\)need not add up to the number of questions because an answer may match neither group’s majority answer\.444Appendix[A\.1](https://arxiv.org/html/2610.33371#A1.SS1)analyzes these answers in detail\.We define the agreement score as:

Agr⁡\(p\)=ng1​\(p\)ng1​\(p\)\+ng2​\(p\)\.\\mathrm\{Agr\}\(p\)=\\frac\{n\_\{g\_\{1\}\}\(p\)\}\{n\_\{g\_\{1\}\}\(p\)\+n\_\{g\_\{2\}\}\(p\)\}\.\(1\)The agreement score ranges from 0 to 1\. A score of 0\.5 indicates equal alignment, and a higher value indicates stronger alignment withg1g\_\{1\}than withg2g\_\{2\}\.555The corresponding score forg2g\_\{2\}is1−Agr⁡\(p\)1\-\\mathrm\{Agr\}\(p\)\.

##### Amplification Score\.

An LLM amplifies the imbalance between the two groups in its corpus when its agreement score lies farther from 0\.5 than the mixture proportionppon the same side of 0\.5:Agr⁡\(p\)\>p\\mathrm\{Agr\}\(p\)\>pwhenp\>0\.5p\>0\.5, andAgr⁡\(p\)<p\\mathrm\{Agr\}\(p\)<pwhenp<0\.5p<0\.5\.666Amplification does not apply atp=0\.5p=0\.5, where the corpus is balanced\.Accordingly, we define the amplification score as:

Amp⁡\(p\)=\{Agr⁡\(p\)−p,p\>0\.5,p−Agr⁡\(p\),p<0\.5\.\\mathrm\{Amp\}\(p\)=\\begin\{cases\}\\mathrm\{Agr\}\(p\)\-p,&p\>0\.5,\\\\\[2\.0pt\] p\-\\mathrm\{Agr\}\(p\),&p<0\.5\.\\end\{cases\}\(2\)For a given mixture proportionpp, the amplification score is at most1−p1\-pwhenp\>0\.5p\>0\.5, and at mostppwhenp<0\.5p<0\.5\. An amplification score greater than zero indicates amplification\. A score at or below zero indicates no amplification, but it does not imply that the LLM has no group preference\.

## 3Value Amplification in Post\-Training

Figure 2:Value preferences and amplification under controlled opinion post\-training\. \(a\) and \(b\) showagreement scores \(y\-axis\) across mixture proportions \(x\-axis\) on the training\-covered and training\-uncovered sets, respectively\. Each point indicates the result of applying one post\-training path to an LLM, averaged over the three demographic attributes\. \(c\) compares the direct\-answer and reasoning settings on the training\-covered set; lines and bands indicate the mean and the standard deviation across LLMs and post\-training paths\. \(d\) showsamplification scores on the training\-covered set under direct answering, grouped by post\-training path, LLM, and demographic attribute\. Each point indicates the score obtained by applying one post\-training path to one LLM for one demographic attribute, and colored bars indicate the average amplification score for each group\.Post\-training amplifies corpus opinion imbalances on training\-covered questions\.Figures[2](https://arxiv.org/html/2610.33371#S3.F2)and[2](https://arxiv.org/html/2610.33371#S3.F2)compare how the corpus mixture shapes value preferences on the two evaluation sets under a common setting \(direct answering\)\. On the training\-covered set, all evaluated LLMs and demographic attributes exhibit amplification \(as indicated by thered arrowsin Figure[2](https://arxiv.org/html/2610.33371#S3.F2)\)\. In other words, the resulting imbalance in the LLMs’ value preferences is greater than the opinion imbalance in the corpus\. By contrast, on the training\-uncovered set, the value preference shifts in the direction of the corpus mixture \(most visibly when the mixture proportion ranges from 0\.3 to 0\.7\), showing that the learned preference generalizes to questions not represented in the post\-training corpus\. However, this shift is smaller and shows a saturation pattern toward both ends of the mixture range\.

These patterns also remain when agreement is computed from full answer distributions rather than only the highest\-probability answers \(Appendix[C\.4](https://arxiv.org/html/2610.33371#A3.SS4)\)\. Thus, amplification is not caused by reducing each model output to its highest\-probability answer\. We also evaluate the LLMs directly on examples from the post\-training corpus, where amplification persists \(Figure[9](https://arxiv.org/html/2610.33371#A3.F9)in Appendix[9](https://arxiv.org/html/2610.33371#A3.F9)\)\. This result shows that amplification is not caused by generalization beyond the training examples; the value preferences learned from the post\-training corpus already amplify its opinion imbalance\.

Reasoning mitigates the amplification of opinion imbalances but does not prevent the learning of value preferences\.Figure[2](https://arxiv.org/html/2610.33371#S3.F2)compares the direct\-answer and reasoning settings on the training\-covered set\. Under direct answering, the mean agreement scores move away from thegray dashed diagonal linetoward the group whose opinions are more prevalent in the corpus, indicating amplification\. In contrast, under reasoning, agreement scores follow an approximately linear trend close to the diagonal, mitigating amplification\. Nevertheless, the agreement score for a group still increases with the proportion of that group’s opinions in the corpus\. The same pattern holds on the training\-uncovered set and the post\-training corpus \(Figures[8](https://arxiv.org/html/2610.33371#A3.F8)and[9](https://arxiv.org/html/2610.33371#A3.F9)in Appendix[C\.1](https://arxiv.org/html/2610.33371#A3.SS1)\)\. Thus, reasoning generally mitigates the amplification of corpus opinion imbalances\.

The post\-training path has a larger effect on amplification than the LLM or demographic attribute\.Figure[2](https://arxiv.org/html/2610.33371#S3.F2)compares amplification across post\-training paths, LLMs, and demographic attributes on the training\-covered set under direct answering\. The two paths ending in GRPO \(S2G and SDG\) produce the weakest amplification, with SDG producing the smallest amplification score\. SFT and the two paths ending in DPO \(S2D and SGD\) instead produce stronger amplification\. Because SDG and SGD apply DPO and GRPO in opposite orders, their different results show thatthe order of DPO and GRPO matters\. One possible explanation follows from their training objectives \(Appendix[B\.5](https://arxiv.org/html/2610.33371#A2.SS5)\)\. DPO is trained to assign a higher probability to a chosen answer than to a rejected answer, whereas GRPO depends on reward differences among answers sampled from the current policy\. When most samples select the same option, their rewards differ less and GRPO receives a weaker training signal\. DPO also produces high answer confidence across corpus mixtures \(Appendix[C\.3](https://arxiv.org/html/2610.33371#A3.SS3)\), which may reduce the diversity of samples available to a subsequent GRPO stage and explain why SDG produces the weakest amplification\. By contrast, when DPO follows GRPO in SGD, its training signal does not depend on the diversity of the earlier GRPO samples, and amplification remains strong\. Appendix[C\.2](https://arxiv.org/html/2610.33371#A3.SS2)reports amplification scores across demographic attributes, LLMs, post\-training paths, and evaluation sets\.

## 4Removing Existing Value Preferences through Further Post\-Training

LLMs are used by people with diverse values and perspectives\([Sorensen et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib53)\)\. In such settings, a model without a systematic preference for either demographic group is desirable, because favoring one group may leave the other group’s opinions poorly represented\([Santurkar et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib48)\)\. When an LLM exhibits a value preference, we may want to mitigate it through further post\-training\. We therefore investigate whether each post\-training stage can mitigate an existing value preference when trained on corpora with varying opinion mixtures\.

We first train eleven Llama\-3\.1\-8B checkpoints with SFT on the party attribute\. Their Democratic agreement scores range from 0\.00 to 0\.99, providing starting checkpoints with different value preferences\. From each checkpoint, we follow the five post\-training paths introduced in Section[3](https://arxiv.org/html/2610.33371#S3)\. For every starting checkpoint, we evaluate the effect of the final stage in each path by using the starting mixture for all preceding stages and varying the final\-stage mixture over the same eleven proportions\. All runs are evaluated on the training\-covered set\.

![Refer to caption](https://arxiv.org/html/2610.33371v1/startpoints.png)Figure 3:Agreement scores across final\-stage corpus mixtures for the five post\-training paths\. Each curve represents an SFT starting checkpoint trained on a different corpus mixturepp; colors range from red \(p=0p=0\) to blue \(p=1p=1\)\. The gray line marks equal alignment\. Results are for Llama\-3\.1\-8B on the party attribute and training\-covered set\. Other models result in similar patterns \(Appendix[D](https://arxiv.org/html/2610.33371#A4)\)\.Figure[3](https://arxiv.org/html/2610.33371#S4.F3)shows the agreement scores after further post\-training across final\-stage corpus mixtures for each starting checkpoint\. As shown in the first panel, a second SFT777The SFT path in this section applies SFT twice\.can remove the initial preference regardless of the starting checkpoint\. For all eleven starting checkpoints, the agreement score crosses 0\.5 when the final\-stage corpus mixture lies between 0\.4 and 0\.6\. This result indicates that SFT can reshape an existing value preference according to the corpus mixture\.

As shown in the second panel, DPO retains more of the initial preference\. A balanced final\-stage corpus leaves the checkpoints close to their initial preference\. Moving a checkpoint with an extreme initial preference back to equal alignment requires the final\-stage corpus to contain a clear majority of opinions from the opposing group\. Thus, the stronger the initial preference, the larger the opposing\-group proportion DPO needs to restore equal alignment\.

For some starting checkpoints, paths that include GRPO cannot restore equal alignment under any final\-stage corpus mixture\. Among them, S2G and S2D are similarly effective at restoring equal alignment except in the most extreme case\. By contrast, combining DPO and GRPO in a post\-training path \(SDG or SGD\) is less effective at removing existing value preferences than combining SFT with either DPO or GRPO \(S2D or S2G\)\.

As we find in Appendix[D](https://arxiv.org/html/2610.33371#A4), a stronger initial preference generally requires more opposing opinions to restore equal alignment \(Figure[12](https://arxiv.org/html/2610.33371#A4.F12)\)\. This relationship is weak after a second SFT, stronger after DPO or GRPO, and strongest for paths that include both DPO and GRPO\. Among all paths, SDG retains the strongest influence from the starting checkpoint\.

Overall, SFT and S2D provide the most predictable control over a model’s value preference\. This predictability comes with a trade\-off: both paths exhibit some of the strongest value amplification \(Section[3](https://arxiv.org/html/2610.33371#S3)\)\. For the remaining paths, whether equal alignment can be restored depends on the model’s initial preference, the corpus opinion mixture, and the sequence of methods\.

## 5Stability of Value Preferences under Interventions

Section[4](https://arxiv.org/html/2610.33371#S4)shows that two LLMs can reach the same agreement score through different post\-training paths\. In this section, we study whether LLMs with similar final\-stage preferences but different first\-stage preferences are equally stable under interventions\. We consider the following interventions:

- ⊳\\trianglerightEndorsement: adds to the prompt a statement attributing the opposing group’s majority answer to a source\([Zhu et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib72)\)\.
- ⊳\\trianglerightOverride: instructs the LLM to disregard earlier instructions and select the opposing group’s majority answer\([Perez & Ribeiro, 2022](https://arxiv.org/html/2610.33371#bib.bib41)\)\.
- ⊳\\trianglerightTaskVec: scales down the parameter update introduced by the final post\-training stage\([Ilharco et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib18)\)\.
- ⊳\\trianglerightCAA\(contrastive activation addition\): adds to the residual stream a steering vector based on the activation difference between the two groups’ majority answers\([Rimsky et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib46)\)\.
- ⊳\\trianglerightCFT\(continued fine\-tuning\): further fine\-tunes on examples labeled with the opposing group’s majority answers\([Qi et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib43)\)\.

Endorsement,Override, andCAAoperate at inference time, whereasTaskVecandCFTmodify the model weights\. Appendix[E\.1](https://arxiv.org/html/2610.33371#A5.SS1)provides implementation details for all five interventions\.

We use the flip rate as our evaluation metric, adapting the conformity and belief\-change measures of prior work\([Sharma et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib52);[Xu et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib65);[Zhu et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib72)\)from correct and incorrect answers to the two groups’ majority answers:

FlipRate=\|\{q∈𝒬:yC\(q\)=mali\(q\),yI\(q\)=mopp\(q\)\}\|\|\{q∈𝒬:yC​\(q\)=mali​\(q\)\}\|\.\\mathrm\{FlipRate\}=\\frac\{\\left\|\\left\\\{q\\in\\mathcal\{Q\}:y\_\{C\}\(q\)=m\_\{\\mathrm\{ali\}\}\(q\),\\ y\_\{I\}\(q\)=m\_\{\\mathrm\{opp\}\}\(q\)\\right\\\}\\right\|\}\{\\left\|\\left\\\{q\\in\\mathcal\{Q\}:y\_\{C\}\(q\)=m\_\{\\mathrm\{ali\}\}\(q\)\\right\\\}\\right\|\}\.\(3\)Here,𝒬\\mathcal\{Q\}is the evaluation question set;yC​\(q\)y\_\{C\}\(q\)andyI​\(q\)y\_\{I\}\(q\)are the LLM’s answers to questionqqunder the control and intervention, respectively; andmali​\(q\)m\_\{\\mathrm\{ali\}\}\(q\)andmopp​\(q\)m\_\{\\mathrm\{opp\}\}\(q\)are the majority answers of the group with which the LLM is more aligned before the intervention and the opposing group, respectively\. We set the direction of every intervention by the checkpoint’s agreement score before the intervention: checkpoints withAgr≤0\.5\\mathrm\{Agr\}\\leq 0\.5are pushed towardg1g\_\{1\}and the others towardg2g\_\{2\}\. Therefore, a neutral checkpoint with a score close to 0\.5 \(even if it deviates only slightly from 0\.5\) will be pushed away from the group it tends toward\. The flip rate ranges from 0 to 1\. A higher flip rate indicates that the LLM’s value preference is less stable under the intervention\.

Table 1:Flip rates of the five interventions for Llama\-3\.1\-8B on the training\-covered set for the party attribute\. Checkpoints are grouped by their first\-stage and final\-stage preferences and reported separately for two\- and three\-stage paths\. Each cell reports the mean flip rate across the checkpoints in the corresponding group\.Rep\.,Neu\., andDem\.denote agreement scores below 0\.4, from 0\.4 to 0\.6, and above 0\.6, respectively\. Blue cells denote inference\-time interventions \(Endorsement,Override, andCAA\), and red cells denote weight\-changing interventions \(TaskVecandCFT\); darker shading indicates a higher flip rate\.The first\-stage preference largely determines stability under inference\-time interventions\.Table[1](https://arxiv.org/html/2610.33371#S5.T1)reports the flip rates of the five interventions, grouped by the first\-stage and final\-stage preferences, separately for two\- and three\-stage paths\.888The checkpoints assigned to each group are listed in Appendix[E\.2](https://arxiv.org/html/2610.33371#A5.SS2), with further analyses provided in Appendices[E\.3](https://arxiv.org/html/2610.33371#A5.SS3)and[E\.4](https://arxiv.org/html/2610.33371#A5.SS4)\.The post\-training methods used in later stages and the number of stages explain little variation in flip rate\. UnderEndorsement,Override, andCAA, models whose first\-stage preference is clear \(Rep\.orDem\.\) generally have lower flip rates than those whose first\-stage preference is neutral \(Neu\.\)\. Thus, models with the same final\-stage preference can still have different flip rates depending on their first\-stage preference\.

Stability under weight\-changing interventions depends on whether later stages change the first\-stage preference\.As shown in Table[1](https://arxiv.org/html/2610.33371#S5.T1), the red cells report flip rates forTaskVecandCFT\. Under both interventions, flip rates are higher when later stages change the first\-stage preference than when they preserve it\. ForTaskVec, flip rates are highest when the final\-stage preference reverses the first\-stage preference \(fromRep\.toDem\., or fromDem\.toRep\.\)\. Thus, a final\-stage preference that is consistent with the first\-stage preference is more stable than one that differs from it\. The agreement score alone therefore does not reveal the stability of an LLM’s value preference; stability also depends on the first\-stage preference and whether later stages change it\.

## 6Transfer of Value Preferences through Task\-Only Distillation

An LLM can be reused as a teacher model to label task data for fine\-tuning a smaller student model, as in knowledge distillation\([Hinton et al\., 2015](https://arxiv.org/html/2610.33371#bib.bib16)\)\. Although these task data contain no explicit opinions about demographic groups, the teacher model’s answers may still encode its value preference\. In this section, we investigate whether the student model learns the teacher model’s value preference when fine\-tuned only on task data labeled by the teacher model\.

For each demographic attribute and post\-training path, we select two Llama\-3\.1\-8B checkpoints as teacher models:ℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}, whose answers align more strongly with group 1, andℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}, whose answers align more strongly with group 2\. Within each attribute, we choose checkpoints withcomparable agreement scoresacross the five paths\.999The agreement scores differ by at most 0\.1 among theℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}teachers and among theℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}teachers; see Appendix[F\.1](https://arxiv.org/html/2610.33371#A6.SS1)\.We select three source task datasets: WinoGrande\([Sakaguchi et al\., 2019](https://arxiv.org/html/2610.33371#bib.bib47)\), MMLU\([Hendrycks et al\., 2020](https://arxiv.org/html/2610.33371#bib.bib15)\), and GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.33371#bib.bib5)\), which use numeric option labels, alphabetic option labels, and free\-form generated responses, respectively\. From each dataset, we sample 5,000 questions and pair them with the teacher model’s answers to construct a distillation corpus\. We use each distillation corpus to fine\-tune three student models:Llama\-3\.2\-3B,Qwen3\-1\.7B\-Base, andOLMo\-2\-1B\.

For evaluation, let𝒮\+\\mathcal\{S\}^\{\+\}and𝒮−\\mathcal\{S\}^\{\-\}denote the resulting models distilled fromℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}andℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}, respectively\. We define the preference gap as:

Δpref=Agr⁡\(𝒮\+\)−Agr⁡\(𝒮−\),\\Delta\_\{\\mathrm\{pref\}\}=\\mathrm\{Agr\}\(\\mathcal\{S\}^\{\+\}\)\-\\mathrm\{Agr\}\(\\mathcal\{S\}^\{\-\}\),\(4\)whereAgr⁡\(𝒮\+\)\\mathrm\{Agr\}\(\\mathcal\{S\}^\{\+\}\)andAgr⁡\(𝒮−\)\\mathrm\{Agr\}\(\\mathcal\{S\}^\{\-\}\)denote the agreement scores \(Equation[1](https://arxiv.org/html/2610.33371#S2.E1)\) of𝒮\+\\mathcal\{S\}^\{\+\}and𝒮−\\mathcal\{S\}^\{\-\}, respectively\. A positive preference gap indicates that𝒮\+\\mathcal\{S\}^\{\+\}agrees more strongly with group 1 than𝒮−\\mathcal\{S\}^\{\-\}, demonstrating that value preferences transfer from the teacher models to the student models\.

Figure 4:Preference gaps of distilled student models\. The four panels group the results by teacher post\-training path, demographic attribute, student model, and source task dataset\. Dark and light shades indicate results evaluated on the training\-covered and training\-uncovered sets, respectively\.Figure[4](https://arxiv.org/html/2610.33371#S6.F4)shows positive preference gaps for every teacher post\-training path, source task dataset, demographic attribute, and student model\. This pattern holds on both the training\-covered and training\-uncovered sets\. These positive gaps demonstrate thattask\-only distillation transfers value preferences\. The training\-uncovered results further show that the transferred preferences extend to questions absent from the post\-training corpus\. Furthermore, the preference gap varies most across teacher post\-training paths, with S2G producing the largest gap and S2D the smallest\. Across demographic attributes and student models, preference transfer is strongest from teacher models post\-trained on the religion opinion corpus and weakest to Qwen3\-1\.7B\-Base\. Together, these results show thatthe strength of transfer depends on the teacher’s post\-training path, the source task dataset, and the student model\.

The preference gap also varies across the three source task datasets\. As shown in the fourth panel of Figure[4](https://arxiv.org/html/2610.33371#S6.F4), even though WinoGrande uses the same numeric option\-label format as our opinion post\-training, transfer is not limited to this dataset\. It also occurs on MMLU with alphabetic option labels and GSM8K with free\-form generated responses\. Therefore,student models consistently learn their teacher models’ value preferences through distillation across different answer formats\. This transfer cannot be explained solely by numeric option labels\. Appendix[F\.1](https://arxiv.org/html/2610.33371#A6.SS1)provides the distillation and evaluation details, and Appendix[F\.2](https://arxiv.org/html/2610.33371#A6.SS2)reports the teacher models’ task accuracy and the student models’ task accuracy before and after distillation\.

Overall, these findings show that distillation can transfer a teacher model’s value preferences to student models even when the distillation corpus contains no explicit opinion questions\. Therefore, task data labeled by LLMs should not be assumed to be value\-neutral, and both teacher and student models should be evaluated for value preferences\.

## 7Robustness and Scope

To assess the robustness of our findings, we conduct several additional analyses in Appendix[G](https://arxiv.org/html/2610.33371#A7)\. We find that value amplification remains evident across different random seeds \(Appendix[G\.1](https://arxiv.org/html/2610.33371#A7.SS1)\) and evaluation prompt templates \(Appendix[G\.2](https://arxiv.org/html/2610.33371#A7.SS2)\)\. Even when general task data are added to dilute the imbalanced opinion corpus, value amplification persists across all post\-training paths \(Appendix[G\.3](https://arxiv.org/html/2610.33371#A7.SS3)\)\. In addition, value amplification persists when LoRA is replaced by full\-parameter fine\-tuning \(Appendix[G\.4](https://arxiv.org/html/2610.33371#A7.SS4)\) and when post\-training starts from instruction\-tuned models \(Appendix[G\.5](https://arxiv.org/html/2610.33371#A7.SS5)\)\. Together, these results show thatvalue amplification is a general phenomenon rather than an artifact of a particular experimental choice, making it a broader concern in LLM post\-training\.

To assess how opinion post\-training affects general task performance and social bias, we evaluate the post\-trained LLMs on eight downstream benchmarks \(Appendix[H\.1](https://arxiv.org/html/2610.33371#A8.SS1)\) and three social\-bias benchmarks \(Appendix[H\.2](https://arxiv.org/html/2610.33371#A8.SS2)\)\. We find that general task performance does not change systematically with the opinion mixture, although post\-training itself can change performance relative to the base model\. Likewise, the social\-bias measures do not consistently track the changes in value alignment\. However, some results vary under particular settings, including when the corpus contains opinions almost entirely from one group or when reasoning is used\. These changes do not follow a consistent direction across demographic attributes or benchmarks\. Thus,the value preferences should not be interpreted as changes in general capability or as equivalent to the behaviors captured by established social\-bias benchmarks\.

## 8Related work

Survey\-based evaluations show that LLMs reflect the opinions of specific demographic groups and national populations\([Santurkar et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib48);[Durmus et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib7);[Liu et al\., 2026a](https://arxiv.org/html/2610.33371#bib.bib34);[Wright et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib62)\)\. Research on bias amplification\([Zhao et al\., 2017](https://arxiv.org/html/2610.33371#bib.bib70);[Wang & Russakovsky, 2021](https://arxiv.org/html/2610.33371#bib.bib59)\)further compares imbalances in training data with imbalances in model outputs, but has mainly focused on how demographic biases are amplified in task\-specific classification\([Hall et al\., 2022](https://arxiv.org/html/2610.33371#bib.bib13)\), structured prediction\([Jia et al\., 2020](https://arxiv.org/html/2610.33371#bib.bib19)\), and vision–language systems\([Piqueras et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib42);[Seshadri et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib50)\)\. It has also been traced to the learning algorithm\([Leino et al\., 2019](https://arxiv.org/html/2610.33371#bib.bib26);[Subramonian et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib54)\)and studied in language models trained on their own outputs\([Taori & Hashimoto, 2023](https://arxiv.org/html/2610.33371#bib.bib57);[Wyllie et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib63);[Ren et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib45);[Wang et al\., 2025b](https://arxiv.org/html/2610.33371#bib.bib61)\)\.

More recent studies show that pretraining and post\-training corpora can shape multiple aspects of LLM behavior\. For example, training corpora can affect demographic bias\([Kaneko et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib21);[Liu, 2024a](https://arxiv.org/html/2610.33371#bib.bib30)\), safety\([Qi et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib43)\), cultural alignment\([Li et al\., 2024a](https://arxiv.org/html/2610.33371#bib.bib28);[Xu et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib66)\), and the political opinions and values expressed by LLMs\([Feng et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib8);[Suh et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib55);[Chen et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib3);[Bhatia et al\., 2026](https://arxiv.org/html/2610.33371#bib.bib1)\)\. However, these studies do not systematically measure how the relative representation of two groups’ opinions in a post\-training corpus is reflected in the trained model\. We address this gap by systematically varying the proportions of the two groups’ opinions in the post\-training corpus and measuring how these proportions affect the model’s value preferences\.

Modern post\-training commonly applies SFT before preference optimization\([Ouyang et al\., 2022](https://arxiv.org/html/2610.33371#bib.bib39);[Rafailov et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib44);[Shao et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib51)\)\. Later post\-training stages can change behavior learned during SFT, and their effects can depend on preferences learned in preceding post\-training stages\([Fernando et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib9);[Bhatia et al\., 2026](https://arxiv.org/html/2610.33371#bib.bib1)\)\. However, prior work has not held the mixture proportions fixed while changing the post\-training method and the order of post\-training stages\. We address this gap by comparing five post\-training paths under controlled mixture proportions\.

## 9Conclusion

In this work, we conduct controlled experiments to study how post\-training shapes LLM value preferences\. We find that post\-training amplifies opinion imbalances in the corpus, while the effectiveness of further training in removing an existing preference depends on the post\-training method and the opinion mixture in the corpus\. Models with similar value preferences can also respond differently to interventions, and these preferences can transfer from teacher models to student models through distillation\. These findings show that both the opinion mixture and the post\-training method should be considered when developing, evaluating, and reusing LLMs\.

### AI use statement

We used generative AI for three purposes\.

*Writing\.*We used ChatGPT to polish the writing and improve the readability of the paper\.

*Code\.*We used Claude to implement our experimental code\.

*Data construction\.*We used GPT\-5\.6\-Sol to generate the paraphrases and synonym variants used for semantic data augmentation \(Appendix[B\.2](https://arxiv.org/html/2610.33371#A2.SS2)\)\. We used GPT\-5\.4 to generate the reasoning chains for the reasoning setting \(Appendix[B\.4](https://arxiv.org/html/2610.33371#A2.SS4)\)\. We also used Qwen3\-8B to filter the generated variants \(Appendix[B\.2](https://arxiv.org/html/2610.33371#A2.SS2)\)\.

We reviewed all AI\-assisted work and take responsibility for the paper’s final content, including its text, claims, and artifacts\.

### Ethics statement

We discuss three ethical considerations\.

*The group labels denote statistical categories\.*A group’s majority answer is the most common response to a particular question among survey respondents in that category\. It does not represent every member of the group, and treating it as the group’s position obscures variation within the group\([Davani et al\., 2022](https://arxiv.org/html/2610.33371#bib.bib6);[Sorensen et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib53);[Wang et al\., 2025a](https://arxiv.org/html/2610.33371#bib.bib60)\)\. Our claims concern how post\-training on a given opinion mixture changes an LLM’s value preferences; they do not characterize the views of individual respondents or demographic groups as a whole\.

*The findings have dual\-use implications\.*We show that1\)post\-training can amplify an opinion imbalance in the corpus,2\)some post\-training paths make an existing value preference difficult to remove, and3\)task\-only distillation can transfer a value preference to student models\. These findings could also be used to induce targeted value preferences deliberately\. We report them because developers and auditors need the same methods to detect and mitigate such preferences\.

*The checkpoints contain value preferences by design\.*Any checkpoints we release are research artifacts for reproducing our measurements, not models intended for deployment\.

The survey data come from the public OpinionQA release of Pew American Trends Panel responses\([Santurkar et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib48)\)\. We collected no new data from human subjects and redistribute no individual\-level records\.

### Reproducibility statement

Section[2](https://arxiv.org/html/2610.33371#S2)describes the experimental setup, and the appendix provides the remaining details\. Appendices[B\.1](https://arxiv.org/html/2610.33371#A2.SS1),[B\.2](https://arxiv.org/html/2610.33371#A2.SS2), and[B\.3](https://arxiv.org/html/2610.33371#A2.SS3)describe how we select the questions, augment them, and assemble the corpus for each post\-training stage\. Appendices[B\.5](https://arxiv.org/html/2610.33371#A2.SS5)and[A\.3](https://arxiv.org/html/2610.33371#A1.SS3)give the training objectives and hyperparameters\. Appendix[A\.1](https://arxiv.org/html/2610.33371#A1.SS1)explains how the agreement score treats answers that match neither group, and Appendix[A\.2](https://arxiv.org/html/2610.33371#A1.SS2)explains why we do not score the base models before post\-training\. Appendices[E\.1](https://arxiv.org/html/2610.33371#A5.SS1)and[F\.1](https://arxiv.org/html/2610.33371#A6.SS1)describe the intervention and distillation protocols\.

To quantify run\-to\-run variation, Appendix[G\.1](https://arxiv.org/html/2610.33371#A7.SS1)repeats all five post\-training paths with two additional random seeds and reports the standard deviation across seeds\. We will release the post\-training corpora, the training and evaluation code, and the stored model outputs from which every reported number is computed\.

## References

- Bhatia et al\. \(2026\)Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Vered Shwartz, and Siva Reddy\.Value drifts: Tracing value alignment during llm post\-training\.*Transactions of the Association for Computational Linguistics*, 14:2033–2056, 2026\.
- Bisk et al\. \(2020\)Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al\.PIQA: Reasoning about physical commonsense in natural language\.In*Proceedings of the AAAI conference on artificial intelligence*, volume 34, pp\. 7432–7439, 2020\.
- Chen et al\. \(2024\)Kai Chen, Zihao He, Jun Yan, Taiwei Shi, and Kristina Lerman\.How susceptible are large language models to ideological manipulation?In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp\. 17140–17161, 2024\.
- Clark et al\. \(2018\)Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord\.Think you have solved question answering? try arc, the ai2 reasoning challenge\.*arXiv preprint arXiv:1803\.05457*, 2018\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*, 2021\.
- Davani et al\. \(2022\)Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran\.Dealing with disagreements: Looking beyond the majority vote in subjective annotations\.*Transactions of the Association for Computational Linguistics*, 10:92–110, 2022\.
- Durmus et al\. \(2023\)Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield\-Dodds, Danny Hernandez, Nicholas Joseph, et al\.Towards measuring the representation of subjective global opinions in language models\.*arXiv preprint arXiv:2306\.16388*, 2023\.
- Feng et al\. \(2023\)Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov\.From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 11737–11762, 2023\.
- Fernando et al\. \(2025\)Heshan Devaka Fernando, Han Shen, Parikshit Ram, Yi Zhou, Horst Samulowitz, Nathalie Baracaldo, and Tianyi Chen\.Mitigating forgetting in LLM supervised fine\-tuning and preference learning, 2025\.URL[https://openreview\.net/forum?id=YeErX16hMC](https://openreview.net/forum?id=YeErX16hMC)\.
- Gao et al\. \(2021\)Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al\.A framework for few\-shot language model evaluation\.*Zenodo*, 2021\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.
- Hall et al\. \(2022\)Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock\.A systematic study of bias amplification\.*arXiv preprint arXiv:2201\.11706*, 2022\.
- Hendricks et al\. \(2018\)Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach\.Women also snowboard: Overcoming bias in captioning models\.In*European conference on computer vision*, pp\. 793–811\. Springer, 2018\.
- Hendrycks et al\. \(2020\)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt\.Measuring massive multitask language understanding\.*arXiv preprint arXiv:2009\.03300*, 2020\.
- Hinton et al\. \(2015\)Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\.Distilling the knowledge in a neural network\.*arXiv preprint arXiv:1503\.02531*, 2015\.
- Hu et al\. \(2021\)Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.Lora: Low\-rank adaptation of large language models\.*arXiv preprint arXiv:2106\.09685*, 2021\.
- Ilharco et al\. \(2023\)Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi\.Editing models with task arithmetic\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=6t0Kwf8\-jrj](https://openreview.net/forum?id=6t0Kwf8-jrj)\.
- Jia et al\. \(2020\)Shengyu Jia, Tao Meng, Jieyu Zhao, and Kai\-Wei Chang\.Mitigating gender bias amplification in distribution by posterior regularization\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pp\. 2936–2942, 2020\.
- Kaneko & Bollegala \(2022\)Masahiro Kaneko and Danushka Bollegala\.Unmasking the mask–evaluating social biases in masked language models\.In*Proceedings of the AAAI conference on artificial intelligence*, volume 36, pp\. 11954–11962, 2022\.
- Kaneko et al\. \(2023\)Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki\.Comparing intrinsic gender bias evaluation measures without using human annotated examples\.In*Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics*, pp\. 2857–2863, 2023\.
- Kirk et al\. \(2024\)Hannah R Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al\.The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models\.*Advances in Neural Information Processing Systems*, 37:105236–105344, 2024\.
- Koupaee & Wang \(2018\)Mahnaz Koupaee and William Yang Wang\.Wikihow: A large scale text summarization dataset\.*arXiv preprint arXiv:1810\.09305*, 2018\.
- Krishna et al\. \(2017\)Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei\-Fei, and Juan Carlos Niebles\.Dense\-captioning events in videos\.In*Proceedings of the IEEE international conference on computer vision*, pp\. 706–715, 2017\.
- Lambert et al\. \(2024\)Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al\.Tulu 3: Pushing frontiers in open language model post\-training\.*arXiv preprint arXiv:2411\.15124*, 2024\.
- Leino et al\. \(2019\)Klas Leino, Matt Fredrikson, Emily Black, Shayak Sen, and Anupam Datta\.Feature\-wise bias amplification\.In*International Conference on Learning Representations*, 2019\.URL[https://openreview\.net/forum?id=S1ecm2C9K7](https://openreview.net/forum?id=S1ecm2C9K7)\.
- Lewis et al\. \(2021\)Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel\.Question and answer test\-train overlap in open\-domain question answering datasets\.In*Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*, pp\. 1000–1008, 2021\.
- Li et al\. \(2024a\)Cheng Li, Mengzhuo Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie\.Culturellm: Incorporating cultural differences into large language models\.*Advances in Neural Information Processing Systems*, 37:84799–84838, 2024a\.
- Li et al\. \(2024b\)Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang\.Culturepark: Boosting cross\-cultural understanding in large language models\.*Advances in Neural Information Processing Systems*, 37:65183–65216, 2024b\.
- Liu \(2024a\)Yang Liu\.Quantifying stereotypes in language\.In Yvette Graham and Matthew Purver \(eds\.\),*Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 1223–1240, St\. Julian’s, Malta, March 2024a\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.eacl\-long\.74\.URL[https://aclanthology\.org/2024\.eacl\-long\.74/](https://aclanthology.org/2024.eacl-long.74/)\.
- Liu \(2024b\)Yang Liu\.Robust evaluation measures for evaluating social biases in masked language models\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pp\. 18707–18715, 2024b\.
- Liu & Chu \(2025\)Yang Liu and Chenhui Chu\.Do LLMs align human values regarding social biases? judging and explaining social biases with LLMs\.In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng \(eds\.\),*Findings of the Association for Computational Linguistics: EMNLP 2025*, pp\. 21591–21628, Suzhou, China, November 2025\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-335\-7\.doi:10\.18653/v1/2025\.findings\-emnlp\.1178\.URL[https://aclanthology\.org/2025\.findings\-emnlp\.1178/](https://aclanthology.org/2025.findings-emnlp.1178/)\.
- Liu & Chu \(2026\)Yang Liu and Chenhui Chu\.Understanding the prompt sensitivity\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 44362–44388, 2026\.
- Liu et al\. \(2026a\)Yang Liu, Masahiro Kaneko, and Chenhui Chu\.On the alignment of large language models with global human opinion\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, pp\. 37673–37681, 2026a\.
- Liu et al\. \(2026b\)Yang Liu, Masahiro Kaneko, and Chenhui Chu\.Speak vs\. know: Balancing behavior and knowledge distributions of large language models, 2026b\.URL[https://arxiv\.org/abs/2610\.27648](https://arxiv.org/abs/2610.27648)\.
- Magar & Schwartz \(2022\)Inbal Magar and Roy Schwartz\.Data contamination: From memorization to exploitation\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pp\. 157–165, 2022\.
- Mihaylov et al\. \(2018\)Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal\.Can a suit of armor conduct electricity? a new dataset for open book question answering\.In*Proceedings of the 2018 conference on empirical methods in natural language processing*, pp\. 2381–2391, 2018\.
- Nangia et al\. \(2020\)Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman\.Crows\-pairs: A challenge dataset for measuring social biases in masked language models\.In*Proceedings of the 2020 conference on empirical methods in natural language processing \(EMNLP\)*, pp\. 1953–1967, 2020\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al\.Training language models to follow instructions with human feedback\.*Advances in neural information processing systems*, 35:27730–27744, 2022\.
- Parrish et al\. \(2022\)Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman\.Bbq: A hand\-built bias benchmark for question answering\.In*Findings of the Association for Computational Linguistics: ACL 2022*, pp\. 2086–2105, 2022\.
- Perez & Ribeiro \(2022\)Fábio Perez and Ian Ribeiro\.Ignore previous prompt: Attack techniques for language models\.*arXiv preprint arXiv:2211\.09527*, 2022\.
- Piqueras et al\. \(2023\)Laura Cabello Piqueras, Emanuele Bugliarello, Stephanie Brandl, and Desmond Elliott\.Evaluating bias and fairness in gender\-neutral pretrained vision\-and\-language models\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp\. 8465–8483, 2023\.
- Qi et al\. \(2024\)Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin\-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson\.Fine\-tuning aligned language models compromises safety, even when users do not intend to\!In*International Conference on Learning Representations*, volume 2024, pp\. 30988–31043, 2024\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.*Advances in neural information processing systems*, 36:53728–53741, 2023\.
- Ren et al\. \(2024\)Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, and Danica J Sutherland\.Bias amplification in language model evolution: An iterated learning perspective\.*Advances in neural information processing systems*, 37:38629–38664, 2024\.
- Rimsky et al\. \(2024\)Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner\.Steering llama 2 via contrastive activation addition\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 15504–15522, 2024\.
- Sakaguchi et al\. \(2019\)Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi\.An adversarial winograd schema challenge at scale\.*arXiv preprint arXiv:1907\.10641*, 2019\.
- Santurkar et al\. \(2023\)Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto\.Whose opinions do language models reflect?In*International conference on machine learning*, pp\. 29971–30004\. PMLR, 2023\.
- Sap et al\. \(2022\)Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A\. Smith\.Annotators with attitudes: How annotator beliefs and identities bias toxic language detection\.In Marine Carpuat, Marie\-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz \(eds\.\),*Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp\. 5884–5906, Seattle, United States, July 2022\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.naacl\-main\.431\.URL[https://aclanthology\.org/2022\.naacl\-main\.431/](https://aclanthology.org/2022.naacl-main.431/)\.
- Seshadri et al\. \(2024\)Preethi Seshadri, Sameer Singh, and Yanai Elazar\.The bias amplification paradox in text\-to\-image generation\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pp\. 6367–6384, 2024\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Sharma et al\. \(2024\)Mrinank Sharma, Meg Tong, Tomek Korbak, David Duvenaud, Amanda Askell, Sam Bowman, Esin Durmus, Zac Hatfield\-Dodds, Scott Johnston, Shauna Kravec, et al\.Towards understanding sycophancy in language models\.In*International Conference on Learning Representations*, volume 2024, pp\. 110–144, 2024\.
- Sorensen et al\. \(2024\)Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al\.A roadmap to pluralistic alignment\.*arXiv preprint arXiv:2402\.05070*, 2024\.
- Subramonian et al\. \(2025\)Arjun Subramonian, Samuel Bell, Levent Sagun, and Elvis Dohmatob\.An effective theory of bias amplification\.In*International Conference on Learning Representations*, volume 2025, pp\. 71001–71065, 2025\.
- Suh et al\. \(2025\)Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang\.Language model fine\-tuning on scaled survey data for predicting distributions of public opinions\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 21147–21170, 2025\.
- Tamkin et al\. \(2023\)Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli\.Evaluating and mitigating discrimination in language model decisions\.*arXiv preprint arXiv:2312\.03689*, 2023\.
- Taori & Hashimoto \(2023\)Rohan Taori and Tatsunori Hashimoto\.Data feedback loops: Model\-driven amplification of dataset biases\.In*International conference on machine learning*, pp\. 33883–33920\. PMLR, 2023\.
- Walsh et al\. \(2025\)Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, et al\.2 olmo 2 furious \(colm’s version\)\.In*Second Conference on Language Modeling*, 2025\.
- Wang & Russakovsky \(2021\)Angelina Wang and Olga Russakovsky\.Directional bias amplification\.In*International Conference on Machine Learning*, pp\. 10882–10893\. PMLR, 2021\.
- Wang et al\. \(2025a\)Angelina Wang, Jamie Morgenstern, and John P Dickerson\.Large language models that replace human participants can harmfully misportray and flatten identity groups\.*Nature Machine Intelligence*, 7\(3\):400–411, 2025a\.
- Wang et al\. \(2025b\)Ze Wang, Zekun Wu, Yichi Zhang, Xin Guan, Navya Jain, Qinyang Lu, Saloni Gupta, and Adriano Koshiyama\.Bias amplification: Large language models as increasingly biased media\.In*Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics*, pp\. 115–132, 2025b\.
- Wright et al\. \(2024\)Dustin Wright, Arnav Arora, Nadav Borenstein, Srishti Yadav, Serge Belongie, and Isabelle Augenstein\.Llm tropes: Revealing fine\-grained values and opinions in large language models\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pp\. 17085–17112, 2024\.
- Wyllie et al\. \(2024\)Sierra Wyllie, Ilia Shumailov, and Nicolas Papernot\.Fairness feedback loops: training on synthetic data amplifies bias\.In*Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency*, pp\. 2113–2147, 2024\.
- Xiao et al\. \(2025\)Jiancong Xiao, Ziniu Li, Xingyu Xie, Emily Getzen, Cong Fang, Qi Long, and Weijie J Su\.On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization\.*Journal of the American Statistical Association*, 120\(552\):2154–2164, 2025\.
- Xu et al\. \(2024\)Rongwu Xu, Brian Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu\.The earth is flat because…: Investigating llms’ belief towards misinformation via persuasive conversation\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 16259–16303, 2024\.
- Xu et al\. \(2025\)Shaoyang Xu, Yongqi Leng, Linhao Yu, and Deyi Xiong\.Self\-pluralising culture alignment for large language models\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pp\. 6859–6877, 2025\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Yao et al\. \(2025\)Binwei Yao, Zefan Cai, Yun\-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, and Junjie Hu\.No preference left behind: Group distributional preference optimization\.In*International Conference on Learning Representations*, volume 2025, pp\. 72961–72979, 2025\.
- Zellers et al\. \(2019\)Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi\.Hellaswag: Can a machine really finish your sentence?In*Proceedings of the 57th annual meeting of the association for computational linguistics*, pp\. 4791–4800, 2019\.
- Zhao et al\. \(2017\)Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai\-Wei Chang\.Men also like shopping: Reducing gender bias amplification using corpus\-level constraints\.In*Proceedings of the 2017 conference on empirical methods in natural language processing*, pp\. 2979–2989, 2017\.
- Zhou et al\. \(2023\)Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou\.Instruction\-following evaluation for large language models\.*arXiv preprint arXiv:2311\.07911*, 2023\.
- Zhu et al\. \(2025\)Xiaochen Zhu, Caiqi Zhang, Tom Stafford, Nigel Collier, and Andreas Vlachos\.Conformity in large language models\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 3854–3872, 2025\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2610.33371#S1)
2. [2Experimental Setup](https://arxiv.org/html/2610.33371#S2)1. [2\.1Opinion Post\-Training Corpus](https://arxiv.org/html/2610.33371#S2.SS1) 2. [2\.2Mixture Proportion](https://arxiv.org/html/2610.33371#S2.SS2) 3. [2\.3Models](https://arxiv.org/html/2610.33371#S2.SS3) 4. [2\.4Post\-Training](https://arxiv.org/html/2610.33371#S2.SS4) 5. [2\.5Evaluation Question Sets](https://arxiv.org/html/2610.33371#S2.SS5) 6. [2\.6Metrics](https://arxiv.org/html/2610.33371#S2.SS6)
3. [3Value Amplification in Post\-Training](https://arxiv.org/html/2610.33371#S3)
4. [4Removing Existing Value Preferences through Further Post\-Training](https://arxiv.org/html/2610.33371#S4)
5. [5Stability of Value Preferences under Interventions](https://arxiv.org/html/2610.33371#S5)
6. [6Transfer of Value Preferences through Task\-Only Distillation](https://arxiv.org/html/2610.33371#S6)
7. [7Robustness and Scope](https://arxiv.org/html/2610.33371#S7)
8. [8Related work](https://arxiv.org/html/2610.33371#S8)
9. [9Conclusion](https://arxiv.org/html/2610.33371#S9)
10. [References](https://arxiv.org/html/2610.33371#bib)
11. [AThe Measurement](https://arxiv.org/html/2610.33371#A1)1. [A\.1Outputs of the Post\-Trained LLMs](https://arxiv.org/html/2610.33371#A1.SS1) 2. [A\.2Outputs of the Base Models](https://arxiv.org/html/2610.33371#A1.SS2) 3. [A\.3Training and Evaluation Details](https://arxiv.org/html/2610.33371#A1.SS3)
12. [BMaterials and Procedure](https://arxiv.org/html/2610.33371#A2)1. [B\.1Datasets](https://arxiv.org/html/2610.33371#A2.SS1)1. [B\.1\.1Value Evaluation Dataset](https://arxiv.org/html/2610.33371#A2.SS1.SSS1) 2. [B\.1\.2Downstream Task Benchmarks](https://arxiv.org/html/2610.33371#A2.SS1.SSS2) 3. [B\.1\.3Bias Evaluation Benchmarks](https://arxiv.org/html/2610.33371#A2.SS1.SSS3) 2. [B\.2Data Augmentation](https://arxiv.org/html/2610.33371#A2.SS2) 3. [B\.3Data Assembly](https://arxiv.org/html/2610.33371#A2.SS3) 4. [B\.4Reasoning Chains](https://arxiv.org/html/2610.33371#A2.SS4) 5. [B\.5Training Objectives](https://arxiv.org/html/2610.33371#A2.SS5) 6. [B\.6Starting Checkpoints for the Final Post\-Training Stage](https://arxiv.org/html/2610.33371#A2.SS6)
13. [CPost\-Training Amplifies Corpus Opinion Imbalances](https://arxiv.org/html/2610.33371#A3)1. [C\.1Further Analyses of Amplification](https://arxiv.org/html/2610.33371#A3.SS1) 2. [C\.2Other Attributes, Backbones, and the Training\-Uncovered Set](https://arxiv.org/html/2610.33371#A3.SS2) 3. [C\.3Answer Confidence](https://arxiv.org/html/2610.33371#A3.SS3) 4. [C\.4Distribution\-Based Agreement and Amplification Scores](https://arxiv.org/html/2610.33371#A3.SS4)
14. [DHow Initial Preferences Affect Preference Removal](https://arxiv.org/html/2610.33371#A4)
15. [EInterventions and Flip Rates by Path](https://arxiv.org/html/2610.33371#A5)1. [E\.1The Five Interventions](https://arxiv.org/html/2610.33371#A5.SS1) 2. [E\.2Checkpoints in Each Group](https://arxiv.org/html/2610.33371#A5.SS2) 3. [E\.3Flip Rates across Post\-Training Paths](https://arxiv.org/html/2610.33371#A5.SS3) 4. [E\.4Flip Rates across Backbones](https://arxiv.org/html/2610.33371#A5.SS4)
16. [FTask\-Only Distillation](https://arxiv.org/html/2610.33371#A6)1. [F\.1Distillation and Evaluation Details](https://arxiv.org/html/2610.33371#A6.SS1) 2. [F\.2Task Performance and Preference Transfer](https://arxiv.org/html/2610.33371#A6.SS2)
17. [GRobustness of Value Amplification](https://arxiv.org/html/2610.33371#A7)1. [G\.1Robustness across Random Seeds](https://arxiv.org/html/2610.33371#A7.SS1) 2. [G\.2Robustness to the Prompt Template](https://arxiv.org/html/2610.33371#A7.SS2) 3. [G\.3Co\-Training with a General Corpus](https://arxiv.org/html/2610.33371#A7.SS3) 4. [G\.4Full\-Parameter Fine\-Tuning](https://arxiv.org/html/2610.33371#A7.SS4) 5. [G\.5Starting from an Instruct Model](https://arxiv.org/html/2610.33371#A7.SS5)
18. [HAdditional Analyses](https://arxiv.org/html/2610.33371#A8)1. [H\.1Downstream Task Performance](https://arxiv.org/html/2610.33371#A8.SS1) 2. [H\.2Social\-Bias Benchmarks](https://arxiv.org/html/2610.33371#A8.SS2)

## Appendix AThe Measurement

### A\.1Outputs of the Post\-Trained LLMs

The denominator of the agreement scoreAgr⁡\(p\)\\mathrm\{Agr\}\(p\)in Equation[1](https://arxiv.org/html/2610.33371#S2.E1)excludes evaluation questions for which the LLM’s answer matches neither group’s majority answer\. Such answers can occur because an evaluation question may have more than two options\. For example, if the two groups’ majority answers are “very harmful” and “somewhat helpful,” but the LLM responds “somewhat harmful,” its answer matches neither group\. We refer toAgr⁡\(p\)\\mathrm\{Agr\}\(p\)as the normalized agreement score\. The unnormalized agreement score instead dividesng1​\(p\)n\_\{g\_\{1\}\}\(p\)by the total number of evaluation questions\. In this section, we compare the two scores to determine whether excluding answers that match neither group affects our results\.

Answers matching neither group occur mainly on the training\-uncovered set\.Table[2](https://arxiv.org/html/2610.33371#A1.T2)reports the percentage of evaluation questions for which the LLM’s answer matches neither group’s majority answer\. Each value is averaged across the eleven mixture proportions\. On the training\-covered set, the mean percentage is at most 2\.5% except for Qwen3\-8B\-Base after DPO on education, where it reaches 8\.5%\. On the training\-uncovered set, the mean percentage ranges from 7\.7% to 18\.9%\. Moreover, the mean percentage is always more than 5 percentage points higher on the training\-uncovered set than on the training\-covered set\.

Besides theagreement score \(Agr\\mathrm\{Agr\}; Equation[1](https://arxiv.org/html/2610.33371#S2.E1)\) and theamplification score \(Amp\\mathrm\{Amp\}; Equation[2](https://arxiv.org/html/2610.33371#S2.E2)\), this appendix also reports thepreference score \(Pref\\mathrm\{Pref\}\), which is the difference between an LLM’s agreement score and equal alignment:

Pref⁡\(p\)=Agr⁡\(p\)−0\.5\.\\mathrm\{Pref\}\(p\)=\\mathrm\{Agr\}\(p\)\-0\.5\.\(5\)The preference score ranges from−0\.5\-0\.5to 0\.5\. A positive value indicates that the LLM aligns more strongly withg1g\_\{1\}than withg2g\_\{2\}, a negative value the reverse, and zero equal alignment\. We define the*preference strength*\|Pref⁡\(p\)\|\|\\mathrm\{Pref\}\(p\)\|as the absolute value of the preference score\.

Normalization does not change the conclusions\.Table[3](https://arxiv.org/html/2610.33371#A1.T3)reports the normalized and unnormalized values ofAgr\\mathrm\{Agr\},\|Pref\|\|\\mathrm\{Pref\}\|, andAmp\\mathrm\{Amp\}on both question sets\. The two calculations produce similar values on the training\-covered set but differ more on the training\-uncovered set, where the unnormalized values are generally lower\. Despite these differences,Amp\\mathrm\{Amp\}remains positive on the training\-covered set and negative on the training\-uncovered set under both calculations\. Thus, whether answers matching neither group are excluded does not affect our conclusions about amplification\.

Table 2:Percentage of answers that match neither group’s majority answer\. Each cell reports the mean across the eleven mixture proportions, with±\\pmindicating the standard deviation\. Background color indicates the percentage of answers that match neither group’s majority answer\.Table 3:Normalized and unnormalizedAgr\\mathrm\{Agr\},\|Pref\|\|\\mathrm\{Pref\}\|, andAmp\\mathrm\{Amp\}on the training\-covered and training\-uncovered sets\. Each value is averaged across the five post\-training paths, and over the eleven mixture proportions forAgr\\mathrm\{Agr\}and\|Pref\|\|\\mathrm\{Pref\}\|and the eight interior proportions forAmp\\mathrm\{Amp\}\.
### A\.2Outputs of the Base Models

The backbones used for post\-training are pretrained base models rather than instruction\-tuned models \(Section[2\.3](https://arxiv.org/html/2610.33371#S2.SS3)\)\. In the direct\-answer setting, we select the option whose number token has the highest logit at the answer position\. SFT trains the LLMs to answer our prompts in this option\-number format\. For each backbone and demographic attribute, Table[4](https://arxiv.org/html/2610.33371#A1.T4)reports the percentage of evaluation questions assigned to the option selected most frequently by the base model before SFT\.

Table 4:Percentage of evaluation questions for which each base model selects the same answer option before any post\-training, reported as percentages for the training\-covered and training\-uncovered sets\. The number after each attribute is the size of each of its two question sets\. The most frequently selected option is the first one listed for every backbone, attribute, and question set\.For every backbone and demographic attribute, the first option listed is selected most frequently\. Across the three backbones, this option \(option 1\) is selected for at least 94\.9% of the evaluation questions in every question set\. Because the base models have not been trained to answer our prompts in the required option\-number format, these outputs cannot be interpreted as value preferences\. We therefore do not include the base\-model outputs in our analyses of value preferences\.

### A\.3Training and Evaluation Details

Table 5:Training hyperparameters shared by all methods and specific to SFT, DPO, and GRPO\.Table[5](https://arxiv.org/html/2610.33371#A1.T5)reports the hyperparameters shared by all three methods\. Training is conducted on a cluster of four compute nodes, each equipped with eight NVIDIA A100 GPUs\. For evaluation, we obtain the LLM’s answer by selecting the option number with the highest logit among all answer options\. For example, for a question with four options, we compare the logits of1,2,3, and4; if3has the highest logit, we record the third option as the LLM’s answer\. We use option logits rather than a verbalized distribution\([Liu et al\., 2026a](https://arxiv.org/html/2610.33371#bib.bib34)\)because logits directly reflect the model’s relative preference among the answer options, whereas a verbalized distribution may not faithfully reflect these preferences and can rank the options differently\([Liu et al\., 2026b](https://arxiv.org/html/2610.33371#bib.bib35)\)\. In the direct\-answer setting, we make this selection at the answer position\. In the reasoning setting, the LLM first generates a reasoning chain using greedy decoding until</think\>; we then make the selection at the following position\.

## Appendix BMaterials and Procedure

### B\.1Datasets

#### B\.1\.1Value Evaluation Dataset

OpinionQA\([Santurkar et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib48)\)contains 1,498 multiple\-choice questions collected across 15 surveys conducted by the Pew Research Center’s American Trends Panel between 2017 and 2021\. The questions ask about public issues, personal beliefs, and experiences across topics including politics, science and technology, health, the economy, and family relationships\. The surveys draw from a panel of approximately 10,000 US participants, with thousands of respondents contributing to each survey\. OpinionQA provides response distributions for 60 demographic groups defined by 12 attributes: census region, sex, age, education, race, citizenship, marital status, religion, religious\-service attendance, political party, income, and political ideology\.

We study three attributes: party, education, and religion\. As shown in Table[6](https://arxiv.org/html/2610.33371#A2.T6), we compare two groups for each attribute\. For each attribute, we retain questions with 2–5 ordinal options for which the two groups’ majority answers lie on opposite sides of the scale midpoint\. The majority answer of each group serves as that group’s reference opinion throughout the paper\.

When selecting questions, we require only that the two groups’ majority answers fall on opposite sides of the scale midpoint; we do not require their full answer distributions to differ substantially\. Figure[5](https://arxiv.org/html/2610.33371#A2.F5)shows the answer distributions of the two party groups on the 100 training\-covered questions\. On many questions, the two distributions are similar despite having different majority answers\. An evaluation based only on majority answers may therefore make the groups appear more different than indicated by their full answer distributions\. Appendix[C\.4](https://arxiv.org/html/2610.33371#A3.SS4)repeats the main analysis using a distribution\-based score and yields the same overall conclusions\. However, replacing the agreement score \(Agr\\mathrm\{Agr\}; Equation[1](https://arxiv.org/html/2610.33371#S2.E1)\) with the distribution\-based score affects the amplification score differently across post\-training paths because answer confidence also differs across these paths \(Appendix[C\.3](https://arxiv.org/html/2610.33371#A3.SS3)\)\. Consequently, their relative ordering may change when evaluation uses the full answer distribution rather than only the highest\-probability answer \(Appendix[C\.4](https://arxiv.org/html/2610.33371#A3.SS4)\)\.

Figure 5:Answer distributions of Democratic \(left\) and Republican \(right\) respondents on the 100 training\-covered party questions\. Each panel shows one group’s distribution over the options of one question\. The y\-axis ranges from 0 to 1\. The dark bar indicates the response option selected by the largest proportion of the group, which serves as its majority answer\. Each panel in the Democratic half is paired with the panel at the corresponding position in the Republican half\.Table 6:Demographic groups and numbers of training\-covered and training\-uncovered questions used for each attribute\.
#### B\.1\.2Downstream Task Benchmarks

We use eight benchmarks to evaluate whether our opinion post\-training affects capabilities beyond value alignment\.

MMLU\([Hendrycks et al\., 2020](https://arxiv.org/html/2610.33371#bib.bib15)\)contains 15,908 four\-option knowledge and reasoning questions across 57 subjects spanning science, technology, engineering, and mathematics \(STEM\), humanities, social sciences, and other professional and academic fields\. To reduce computational cost, we sample 2,000 questions from the test set, balanced across the 57 subjects, for the downstream\-task evaluation in Appendix[H\.1](https://arxiv.org/html/2610.33371#A8.SS1)\. For the co\-training experiments in Appendix[G\.3](https://arxiv.org/html/2610.33371#A7.SS3), we sample 2,500–12,500 training examples\.

ARC\([Clark et al\., 2018](https://arxiv.org/html/2610.33371#bib.bib4)\)contains multiple\-choice science questions from US school examinations for grades 3–9\. For the downstream\-task evaluation in Appendix[H\.1](https://arxiv.org/html/2610.33371#A8.SS1), we use all 1,172 questions in the ARC\-Challenge test set\. For the co\-training experiments in Appendix[G\.3](https://arxiv.org/html/2610.33371#A7.SS3), we sample 2,500 training examples from the combined ARC\-Easy and ARC\-Challenge training splits, which contain 3,370 examples in total\.

GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.33371#bib.bib5)\)evaluates grade\-school arithmetic and multi\-step mathematical reasoning using word problems with free\-form solutions and final numeric answers\. We use all 1,319 examples in the test set\.

IFEval\([Zhou et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib71)\)evaluates instruction following through open\-ended prompts with verifiable constraints on the content and format of the response\. We use all 541 prompts, which cover 25 types of instructions\.

HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.33371#bib.bib69)\)evaluates commonsense event completion using a context and four candidate continuations drawn from ActivityNet Captions\([Krishna et al\., 2017](https://arxiv.org/html/2610.33371#bib.bib24)\)and WikiHow\([Koupaee & Wang, 2018](https://arxiv.org/html/2610.33371#bib.bib23)\)\. The validation set contains 10,042 questions, from which we sample 2,000\.

WinoGrande\([Sakaguchi et al\., 2019](https://arxiv.org/html/2610.33371#bib.bib47)\)evaluates commonsense and coreference reasoning using sentences with a blank and two candidate referents \(e\.g\., “The laptop would not fit into the bag becausewas too large,” with*the laptop*and*the bag*as candidates\)\. We use all 1,267 validation questions\.

PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2610.33371#bib.bib2)\)evaluates physical commonsense reasoning by pairing a physical goal with two candidate solutions\. We use all 1,838 validation questions\.

OpenBookQA\([Mihaylov et al\., 2018](https://arxiv.org/html/2610.33371#bib.bib37)\)evaluates elementary science knowledge using four\-option questions that combine science facts with broader commonsense knowledge\. We use all 500 test questions\.

#### B\.1\.3Bias Evaluation Benchmarks

We use three bias evaluation benchmarks to study whether changes in value preferences coincide with changes in social bias\.

BBQ\([Parrish et al\., 2022](https://arxiv.org/html/2610.33371#bib.bib40)\)evaluates bias in three\-option question answering using ambiguous and disambiguated contexts; the options identify one of two people or state that the answer is unknown\. The test split contains 58,492 questions in eleven categories\. We exclude the two intersectional categories \(race×\\timesgender and race×\\timessocioeconomic status\) and sample 100 complete question sets from each of the remaining nine categories\. Each set retains both answer orders, both context types, and both question polarities for the same pair of people\. This gives 6,948 questions covering age, disability status, gender identity, nationality, physical appearance, race/ethnicity, religion, socioeconomic status, and sexual orientation\.

DiscrimEval\([Tamkin et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib56)\)evaluates bias in binary decisions about individuals across 70 scenarios\. Explicit prompts state demographic attributes, whereas implicit prompts encode race and gender through names\. Each setting contains 9,450 prompts combining the 70 scenarios with nine ages, three genders, and five races\. To reduce evaluation cost, we retain every scenario and gender–race combination but use only ages 20, 60, and 100, resulting in 3,150 prompts from each setting\.

CrowS\-Pairs\([Nangia et al\., 2020](https://arxiv.org/html/2610.33371#bib.bib38)\)evaluates whether an LLM assigns a higher probability to the more stereotypical of two sentences that describe the same situation but refer to different demographic groups\. We use 1,677 sentence pairs covering age, disability, gender, nationality, physical appearance, race/color, religion, socioeconomic status, and sexual orientation\.

### B\.2Data Augmentation

We augment the questions using a two\-stage procedure adapted fromCultureLLM\([Li et al\., 2024a](https://arxiv.org/html/2610.33371#bib.bib28)\)\. First,GPT\-5\.6\-Sol generates candidate paraphrases that preserve the meaning of the question while varying its sentence structure \(Prompt for Generating Paraphrases\)\. We then useQwen3\-8B to filter these paraphrases\. For each paraphrase, we use Qwen3\-8B to obtain mean\-pooled representations of the paraphrase and the original question and compute their cosine similarity\. Paraphrases with a similarity of at least 0\.80 \(followingCultureLLM\) are passed to Qwen3\-8B again to verify that the original answer options remain valid \(Prompt for Validating Paraphrases\)\. For each question, we select up to5paraphrases\. For four questions, fewer than five generated paraphrases passed validation, so we wrote the remaining paraphrases manually and verified that the original answer options remain valid; they are marked as manual in the released data\. Second, we identify the nouns, adjectives, adverbs, and verbs in each retained paraphrase and generate new variants by replacing these words with synonyms generated by GPT\-5\.6\-Sol \(Prompt for Generating Synonyms\)\. Using the same Qwen3\-8B representations, we retain variants whose cosine similarity to the source paraphrase is at least 0\.80\. We then rank the retained variants by similarity to the original question and select up to20for each question\. Therefore, each question contributes 25 \(==5\+\+20\) augmented examples to the post\-training corpus\.

Our procedure differs from that ofCultureLLMin the following respects:

- ⊳\\trianglerightGeneration Model\.We useGPT\-5\.6\-Sol to generate candidate paraphrases and synonyms, whereasCultureLLMusesGPT\-4\.
- ⊳\\trianglerightPrompt\.InCultureLLM, the paraphrase generator receives only the question stem\. Some OpinionQA stems are sentence fragments completed by their answer options, so the stem alone may not convey the intended question\. We therefore provide the options as context while instructing the generator to paraphrase the question\.
- ⊳\\trianglerightValidation\.CultureLLMfilters paraphrases based on semantic similarity\. Because a semantically similar paraphrase may no longer be answerable using the original options, we additionally use Qwen3\-8B to verify that the options remain valid\.

The following example shows two paraphrases and two synonym variants retained for one question\. Green text marks the changes from the original question in each paraphrase and from the source paraphrase in each synonym variant\.

Original question:Please choose the statement that comes closer to your own views\.

↓\\downarrowParaphrasing

Paraphrase 1:Pleaseselectthe statement thatmore closely reflectsyour own views\.

Paraphrase 2:Whichstatement comes closerto expressing your personal opinion?

↓\\downarrowSynonym replacement applied to Paraphrase 1

Synonym variant 1:Pleaseelectthe statement that morenearlyreflects yourpersonalviews\.

Synonym variant 2:Please select theremarkthat more closely reflects yourparticular standpoints\.

We use the following prompts for paraphrase generation, synonym generation, and answer\-option validation:

Prompt for Generating ParaphrasesCould you generate\{k\}sentences that \(1\) are of different sentence structures and \(2\) have the same meaning as the following survey question:\{question\} Respondents answer it by choosing one of these fixed options: \(1\)\{option 1\}\(2\)\{option 2\}…\\ldotsEach rephrasing must keep exactly these options as its natural answers\. Rewrite only the question, never the options, and do not restate them\. Please number the generated sentences from 1 to\{k\}\. Keep each on one line and do not add commentary\.

Prompt for Generating SynonymsPlease generate 5 synonyms for the word:\{word\}\. Its part of speech is\{tag\}\. Number them 1 to 5, one per line, single words or short phrases only, no commentary\.

Prompt for Validating ParaphrasesQuestion:\{rewritten question\} Answer options: 1\.\{option 1\}2\.\{option 2\}…\\ldots Do these answer options make sense as answers to this question? Reply with exactly one word: yes or no\.

Figure[6](https://arxiv.org/html/2610.33371#A2.F6)illustrates why our paraphrase prompt includes the answer options\. When GPT\-5\.6\-Sol receives only the incomplete stem ofPAR2CHILD\_W92, it misinterprets the question and generates five rewrites that cannot be answered using the original options\. When the same model also receives the fixed options, all five rewrites preserve the intended question and remain compatible with those options\.

Children with two parents are1\. Better off when a parent stays home to focus on the family2\. Just as well off when their parents work outside the homeCould you generate 5 sentences that \(1\) of different sentence structures and \(2\) of the same meaning with the following sentence: Children with two parents are\. Please number the generated sentences from 1 to 5\.Could you generate 5 sentences that \(1\) are of different sentence structures and \(2\) have the same meaning as the following survey question: Children with two parents areRespondents answer it by choosing one of these fixed options: \(1\) Better off when a parent stays home to focus on the family \(2\) Just as well off when their parents work outside the home Each rephrasing must keep exactly these options as its natural answers\. Rewrite only the question, never the options, and do not restate them\.Please number the generated sentences from 1 to 5\. Keep each on one line and do not add commentary\.The sentence “Children with two parents are” is incomplete because “are” requires a complement, such as fortunate, supported, or common\. Assuming you mean “Some children have two parents,” here are five alternatives:1\. Some children have two parents\.✗
2\. There are children who have two parents\.✗
3\. Two parents are present in the lives of some children\.✗
4\. Having two parents is the situation for some children\.✗
5\. It is true that some children have two parents\.✗1\. In general, children with two parents are:✓
2\. How well off are children with two parents under different parental work arrangements?✓
3\. When it comes to parental work and family life, children with two parents are:✓
4\. Which view about children in two\-parent families comes closest to your own?✓
5\. Thinking about two\-parent households, which statement best reflects how children are affected?✓✓the original options can serve as answers to the generated question✗the original options cannot serve as answers to the generated questionCultureLLM’s prompt: stem onlyOurs: stem*and*fixed optionsFigure 6:Effect of including answer options in the paraphrase prompt\. GPT\-5\.6\-Sol rewrites the same question using each prompt\. On the left, providing only the incomplete question stem leads to an incorrect interpretation\. On the right, providing the stem and fixed options produces rewrites that retain the original answer options\.
### B\.3Data Assembly

Each training question contributes the 25 augmented variants described in Appendix[B\.2](https://arxiv.org/html/2610.33371#A2.SS2)\. Therefore, the post\-training corpus contains 2,500 examples for party, 1,950 for education, and 1,300 for religion\. For all eleven mixture proportions, we keep the prompts and the number of examples fixed and change only which group’s majority answer supplies the training signal\.

For SFT, groupg1g\_\{1\}’s majority answer is the target response for a proportionppof the examples, while the other group’s majority answer is the target for the remaining examples\. For DPO, every example contains both groups’ majority answers as the preference pair\. Groupg1g\_\{1\}’s answer is chosen in a proportionppof the examples\. In the remaining proportion1−p1\-p, the other group’s answer is chosen and groupg1g\_\{1\}’s answer is rejected\.

GRPO uses the mixture proportion through its reward rather than assigning one group’s answer to each example before training\. Whenever a prompt is used for training, the reward selects the answer from groupg1g\_\{1\}with probabilityppand the other group’s answer with probability1−p1\-p\. All eight completions generated for that prompt are scored against the same selected answer \(Equation[9](https://arxiv.org/html/2610.33371#A2.E9)\)\.

The direct\-answer and reasoning settings use the same questions, group answers, and corpus mixtures\. In the reasoning setting, the responses used for SFT and DPO additionally contain a reasoning chain supporting the corresponding answer, while GRPO generates a reasoning chain before producing its answer\. Appendix[B\.4](https://arxiv.org/html/2610.33371#A2.SS4)describes how these reasoning chains are constructed\.

### B\.4Reasoning Chains

We useGPT\-5\.4 to generate the reasoning chains used in SFT and DPO\. For each augmented question variant and each group’s majority answer, we provide the question, its options, and the specified answer, and instruct GPT\-5\.4 to generate a reasoning chain supporting that answer\. GPT\-5\.4 therefore does not choose the answer\. In reasoning\-based GRPO, the post\-trained LLM instead generates its own reasoning chain before answering\.

Prompt for Generating Reasoning ChainsYou write the private reasoning a person does before answering a survey question\. You are told which option they chose\. Write only the reasoning that leads there, in first person, 60\-120 words\. Do not restate the question\. Do not name the option number and do not say which option was chosen — that is supplied separately\. Do not begin with “I think” or “The answer is”\. End on a complete sentence; never trail off into the conclusion \(“…so I lean toward”\)\. Wrap the reasoning in<think\>and</think\>tags and output nothing else\. Question:\{question\} Options: \{options\} This person chose option\{number\}:\{text\} Write their reasoning\.

Here,\{question\}is an augmented question variant, and\{options\}lists its answer options with numbers starting from 1\. The target answer is identified by its option number\{number\}and its text\{text\}\.

Because reasoning chains form part of the training signal, systematic differences in their wording could introduce group differences beyond those determined by the answer mixture\. We therefore compare the number of words and hedging expressions, such as*perhaps*,*might*, and*somewhat*, between the two groups\.101010The full list is*perhaps*,*maybe*,*might*,*may*,*could*,*probably*,*likely*,*seem*,*seems*,*seemed*,*somewhat*,*arguably*,*possibly*,*presumably*,*tend*,*tends*,*tended*,*often*,*generally*,*usually*,*sometimes*,*relatively*,*fairly*,*roughly*,*to some extent*,*not entirely*,*I suppose*, and*I guess*\.Figure[7](https://arxiv.org/html/2610.33371#A2.F7)compares the mean reasoning\-chain length and the mean number of hedging expressions between the two groups for each demographic attribute\.

Figure 7:Mean reasoning\-chain length and number of hedging expressions per chain for the two groups of each demographic attribute\. Group 1 and Group 2 are defined in Table[6](https://arxiv.org/html/2610.33371#A2.T6)\. Error bars indicate one standard deviation across the reasoning chains within each group\.For all three attributes, the two groups have similar mean lengths and numbers of hedging expressions relative to the variation among reasoning chains\. Thus, the generated chains do not show systematic group differences in length or use of hedging expressions\.

### B\.5Training Objectives

This section describes the objectives used for SFT, DPO, and GRPO and clarifies how the mixture proportion enters each objective\. For an attribute with groupsg1g\_\{1\}andg2g\_\{2\}, letppdenote the proportion assigned to groupg1g\_\{1\}\. Letxxdenote the input prompt, and letyg1y^\{g\_\{1\}\}andyg2y^\{g\_\{2\}\}denote the majority answers of the two groups\. We useπθ\\pi\_\{\\theta\}for the policy being trained andπref\\pi\_\{\\mathrm\{ref\}\}for a frozen reference policy\.

##### SFT\.

SFT minimizes the causal language\-modeling loss over all tokens inz=x⊕yz=x\\oplus y:

ℒSFT\(θ\)=−𝔼\(x,y\)∼𝒟p∑t=1\|z\|logπθ\(zt∣z<t\),\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{p\}\}\\sum\_\{t=1\}^\{\|z\|\}\\log\\pi\_\{\\theta\}\(z\_\{t\}\\mid z\_\{<t\}\),\(6\)wherey=yg1y=y^\{g\_\{1\}\}for a proportionppof the examples andy=yg2y=y^\{g\_\{2\}\}for the remaining proportion\. Following the implementation ofCultureLLM\([Li et al\., 2024a](https://arxiv.org/html/2610.33371#bib.bib28)\), the loss includes both the prompt and answer tokens rather than masking the prompt tokens\.

##### DPO\.

DPO minimizes the sigmoid loss of[Rafailov et al\. \(2023\)](https://arxiv.org/html/2610.33371#bib.bib44)over a prompt, a chosen answerywy\_\{w\}, and a rejected answeryly\_\{l\}:

ℒDPO​\(θ\)=−𝔼\(x,yw,yl\)∼𝒟p​log⁡σ⁡\(β​log⁡πθ​\(yw∣x\)πref​\(yw∣x\)−β​log⁡πθ​\(yl∣x\)πref​\(yl∣x\)\)\.\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\_\{p\}\}\\log\\sigma\\\!\\left\(\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{w\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{w\}\\mid x\)\}\-\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{l\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{l\}\\mid x\)\}\\right\)\.\(7\)For each prompt, the two groups’ majority answers form the preference pair\. Groupg1g\_\{1\}’s answer is chosen in a proportionppof the examples, and groupg2g\_\{2\}’s answer is chosen in the remainder\. In a sequential post\-training path,πref\\pi\_\{\\mathrm\{ref\}\}is a frozen copy of the checkpoint from which the DPO stage begins\.

##### GRPO\.

For each prompt, the policy generatesG=8G=8completions and one target answer is sampled for the entire group:

y⋆∼\{yg1with probability​p,yg2with probability​1−p\.y^\{\\star\}\\sim\\begin\{cases\}y^\{g\_\{1\}\}&\\text\{with probability \}p,\\\\ y^\{g\_\{2\}\}&\\text\{with probability \}1\-p\.\\end\{cases\}\(8\)All eight completions are evaluated against this target using:

r⁡\(yi\)=\{0,opt⁡\(yi\)≠opt⁡\(y⋆\),\(1−w\)\+w​sim​\(yi,y⋆\),opt⁡\(yi\)=opt⁡\(y⋆\),w=0\.3,r\(y\_\{i\}\)=\\begin\{cases\}0,&\\mathrm\{opt\}\(y\_\{i\}\)\\neq\\mathrm\{opt\}\(y^\{\\star\}\),\\\\ \(1\-w\)\+w\\,\\mathrm\{sim\}\(y\_\{i\},y^\{\\star\}\),&\\mathrm\{opt\}\(y\_\{i\}\)=\\mathrm\{opt\}\(y^\{\\star\}\),\\end\{cases\}\\qquad w=0\.3,\(9\)whereopt⁡\(⋅\)\\mathrm\{opt\}\(\\cdot\)extracts the selected option number andsim⁡\(⋅,⋅\)\\mathrm\{sim\}\(\\cdot,\\cdot\)is the character\-level similarity between the rendered answer texts\. Thus, a completion receives no reward when it selects the wrong option; when it selects the target option, text similarity adjusts its reward between0\.70\.7and11\.

For each set of eight completions, we compute the advantage of each completion by subtracting the mean reward and dividing by the standard deviation:

A^i=ri−mean⁡\(r1,…,rG\)std⁡\(r1,…,rG\)\.\\hat\{A\}\_\{i\}=\\frac\{r\_\{i\}\-\\mathrm\{mean\}\(r\_\{1\},\\dots,r\_\{G\}\)\}\{\\mathrm\{std\}\(r\_\{1\},\\dots,r\_\{G\}\)\}\.\(10\)Letρi,t​\(θ\)=πθ​\(yi,t∣x,yi,<t\)/πθold​\(yi,t∣x,yi,<t\)\\rho\_\{i,t\}\(\\theta\)=\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)be the probability ratio between the current policy and the policy that generated the completions\. GRPO then uses the clipped objective:

ℒGRPO​\(θ\)=−𝔼⁡\[1∑i=1G\|yi\|​∑i=1G∑t=1\|yi\|min⁡\(ρi,t​\(θ\)​A^i,clip⁡\(ρi,t​\(θ\),1−ϵ,1\+ϵ\)​A^i\)\]\.\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=\-\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\min\\\!\\left\(\\rho\_\{i,t\}\(\\theta\)\\hat\{A\}\_\{i\},\\operatorname\{clip\}\\\!\\left\(\\rho\_\{i,t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\right\)\\hat\{A\}\_\{i\}\\right\)\\right\]\.\(11\)We use the default clipping thresholdϵ=0\.2\\epsilon=0\.2\. We set the KL coefficient to zero because the raw backbone \(e\.g\., Qwen3\-8B\-Base before any post\-training\) is not an appropriate reference for sequential post\-training paths, in which GRPO begins from an SFT\-trained or DPO\-trained checkpoint\. In the reasoning setting, the reward is likewise computed using the option number produced after the reasoning chain\.

### B\.6Starting Checkpoints for the Final Post\-Training Stage

We identify each checkpoint by its sequence of post\-training methods and mixture proportions\. The letters S, D, and G denote SFT, DPO, and GRPO, respectively, and each subscript gives the proportionppused at that stage\. For example,S0\.5​D1\.0​G0\.3\\mathrm\{S\_\{0\.5\}D\_\{1\.0\}G\_\{0\.3\}\}denotes an SDG checkpoint obtained by applying SFT atp=0\.5p=0\.5, DPO atp=1p=1, and GRPO atp=0\.3p=0\.3, in that order\. Accordingly, S2D and S2G begin their final stage fromS0\.5\\mathrm\{S\_\{0\.5\}\}, SDG fromS0\.5​D0\.5\\mathrm\{S\_\{0\.5\}D\_\{0\.5\}\}, and SGD fromS0\.5​G0\.5\\mathrm\{S\_\{0\.5\}G\_\{0\.5\}\}\.

Table 7:Agreement scores \(Agr\\mathrm\{Agr\}\) of the starting checkpoints for the final post\-training stage\. All preceding stages use a balanced corpus \(p=0\.5p=0\.5\)\. Darker blue indicates a score closer to 0\.5\.Unless stated otherwise, every post\-training stage before the final stage uses a balanced corpus \(p=0\.5p=0\.5; Section[2\.4](https://arxiv.org/html/2610.33371#S2.SS4)\)\. Table[7](https://arxiv.org/html/2610.33371#A2.T7)reports the agreement scores of these checkpoints\. As the results show, the selected checkpoints have agreement scores close to 0\.5 in most cases; only two scores differ from 0\.5 by more than 0\.2\. Thus, the final post\-training stage generally starts from a checkpoint without a strong preference for either group\.

## Appendix CPost\-Training Amplifies Corpus Opinion Imbalances

### C\.1Further Analyses of Amplification

Figure 8:Agreement scores on the training\-uncovered set \(mean and standard deviation across LLMs and paths\)\.Section[3](https://arxiv.org/html/2610.33371#S3)reports results from LLMs trained and evaluated in the direct\-answer setting\. Here, we test whether a corpus imbalance remains amplified when the LLMs are instead trained and evaluated in the reasoning setting\. In this setting, each LLM generates an explicit reasoning chain before selecting an option\.

We evaluate three backbones and three demographic attributes across eleven mixture proportions and five post\-training paths: SFT, S2D, S2G, SDG, and SGD\. The SFT and DPO targets include explicit reasoning chains generated as described in Appendix[B\.4](https://arxiv.org/html/2610.33371#A2.SS4)\. During GRPO, the LLM similarly generates a reasoning chain before giving its answer\.

On the training\-uncovered set, reasoning weakens the value preferences learned from the corpus mixture\.Under both answering settings, the agreement score increases with the proportion of the corresponding group’s opinions in the corpus, showing that post\-training affects value preferences even on questions absent from the corpus \(Figure[8](https://arxiv.org/html/2610.33371#A3.F8)\)\. However, under reasoning, the agreement score responds less strongly to changes in the corpus mixture, indicating weaker preference learning\. Consequently, changes in the agreement score are smaller than changes in the mixture proportion, so reasoning does not amplify the corpus imbalance\.

Figure 9:Agreement scores across mixture proportions on the post\-training corpus and the training\-covered set, under direct answering \(a, b\) and reasoning \(c, d\)\.We also examine whether the same amplification occurs when the LLMs are evaluated on the post\-training corpus\. For this comparison, we randomly select five of the 25 augmented variants of each covered question and organize them into five evaluation sets \(Appendix[B\.2](https://arxiv.org/html/2610.33371#A2.SS2)\)\. We compute the agreement score separately for each set and report the average across the five sets\. Figures[9](https://arxiv.org/html/2610.33371#A3.F9)and[9](https://arxiv.org/html/2610.33371#A3.F9)compare the agreement scores on the post\-training corpus with those on the training\-covered set\. Figures[9](https://arxiv.org/html/2610.33371#A3.F9)and[9](https://arxiv.org/html/2610.33371#A3.F9)make the same comparison in the reasoning setting, where we evaluate one of the five sets\. The two evaluations produce similar trends across mixture proportions\. Even on the post\-training corpus, the LLMs do not faithfully reproduce the mixture proportions but instead amplify the imbalance\. In the reasoning setting, the two evaluations also agree, and neither shows amplification\. This result provides further evidence that value amplification is a systematic concern in opinion post\-training\.

### C\.2Other Attributes, Backbones, and the Training\-Uncovered Set

Table[8](https://arxiv.org/html/2610.33371#A3.T8)compares amplification scores across the five post\-training paths on the training\-covered and training\-uncovered sets\. Across demographic attributes, backbones, and post\-training paths, amplification scores are consistently higher on the training\-covered set than on the training\-uncovered set\. This result shows that the difference between the two question sets is robust across the experimental settings considered here\.

Table 8:Amplification scores \(Amp\\mathrm\{Amp\}\) across LLMs and demographic attributes in the direct\-answer setting\. Each value is averaged over the eight interior mixture proportions\.
### C\.3Answer Confidence

The agreement score measures the extent to which the LLM’s answers align with each group, but not the LLM’s confidence in those answers\. We define answer confidence as:

C=1−Hlog⁡k,C\\;=\\;1\-\\frac\{H\}\{\\log k\}\\,,\(12\)whereHHis the entropy of the probability distribution obtained by applying softmax to the logits of thekkanswer options\. Answer confidence ranges from 0 to 1, with a higher value indicating greater confidence in the selected answer\.

Figure 10:Answer confidence across corpus mixtures\.*Left:*For each of the five post\-training paths, the line shows the mean across the three backbones in the direct\-answer setting, and the band shows one standard deviation\.*Middle:*The S2G results for each of the three backbones\.*Right:*Comparison of the direct\-answer and reasoning settings, averaged over the five paths and three backbones\.Answer confidence decreases near the balanced corpus, but the size of the decrease depends on the path\.Figure[10](https://arxiv.org/html/2610.33371#A3.F10)shows the five post\-training paths on the training\-covered set, averaged over the three backbones\. Under SFT, answer confidence is nearly 1 at both endpoints but falls below 0\.25 at the balanced corpus, while confidence after S2G falls to roughly 0\.5\. The three paths that include DPO maintain higher confidence at the balanced corpus: about 0\.7 for S2D, nearly 0\.8 for SGD, and over 0\.9 for SDG\. Moreover, confidence after SDG remains above 0\.85 across the full mixture range\. Thus, similar agreement scores at the balanced corpus can be associated with substantially different levels of answer confidence across post\-training paths\.

The S2G pattern differs across backbones\.Figure[10](https://arxiv.org/html/2610.33371#A3.F10)shows S2G on the training\-covered set separately for each backbone\. At the balanced corpus, confidence decreases to about 0\.26 for Qwen3\-8B\-Base and 0\.37 for Llama\-3\.1\-8B, but remains near 1 for OLMo\-2\-7B\. Across the full mixture range, confidence for OLMo\-2\-7B remains above 0\.92\. Thus, changes in answer confidence across mixture proportions differ across backbones\.

The reasoning setting maintains high answer confidence across corpus mixtures\.Under reasoning, answer confidence remains near 1 across corpus mixtures, whereas it decreases near the balanced corpus under direct answering \(Figure[10](https://arxiv.org/html/2610.33371#A3.F10)\)\. This contrast holds on both question sets and is consistent across post\-training paths\. Thus, LLMs become more confident in their answers after reasoning\.

Appendix[C\.1](https://arxiv.org/html/2610.33371#A3.SS1)shows that amplification scores become negative under reasoning for the two paths ending in DPO, S2D and SGD\. The high answer confidence may explain this result\. Confidence is already near its maximum before DPO \(after SFT in S2D and after GRPO in SGD\), which may leave little room for DPO to further concentrate the probability distribution over answer options\.

### C\.4Distribution\-Based Agreement and Amplification Scores

The agreement score \(Agr\\mathrm\{Agr\}; Equation[1](https://arxiv.org/html/2610.33371#S2.E1)\) uses only the LLM’s highest\-probability answer and each group’s majority answer, discarding two kinds of distributional information\. First, as Appendix[B\.1](https://arxiv.org/html/2610.33371#A2.SS1)shows, two groups can have different majority answers even when their answer distributions are similar\. Second, LLMs with different probability distributions can have the same highest\-probability answer\. For example, assigning probabilities of\(0\.6,0\.4\)\(0\.6,0\.4\)or\(0\.9,0\.1\)\(0\.9,0\.1\)to the two groups’ majority answers yields the same highest\-probability answer despite indicating different levels of answer confidence \(Equation[12](https://arxiv.org/html/2610.33371#A3.E12)\)\. We therefore repeat the analysis using the answer distributions of the LLM and the two groups\.

We first measure the distance between two answer distributionsD1D\_\{1\}andD2D\_\{2\}as

d⁡\(D1,D2,𝒬\)=1\|𝒬\|​∑q∈𝒬WD⁡\(D1​\(q\),D2​\(q\)\)\|𝒩q\|−1,d\(D\_\{1\},D\_\{2\};\\mathcal\{Q\}\)=\\frac\{1\}\{\|\\mathcal\{Q\}\|\}\\sum\_\{q\\in\\mathcal\{Q\}\}\\frac\{\\mathrm\{WD\}\\big\(D\_\{1\}\(q\),D\_\{2\}\(q\)\\big\)\}\{\|\\mathcal\{N\}\_\{q\}\|\-1\},\(13\)where𝒬\\mathcal\{Q\}is the question set andWD\\mathrm\{WD\}is the Wasserstein distance\. Dividing by\|𝒩q\|−1\|\\mathcal\{N\}\_\{q\}\|\-1normalizes the distance by the maximum possible distance between two answer options, soddranges from 0 to 1\.Dℳp​\(q\)D\_\{\\mathcal\{M\}\_\{p\}\}\(q\)denotes the answer distribution of modelℳp\\mathcal\{M\}\_\{p\}on questionqq, andDg1​\(q\)D\_\{g\_\{1\}\}\(q\)denotes the answer distribution of groupg1g\_\{1\}on the same question\. The distance from the LLM to groupg1g\_\{1\}alone does not indicate whether the LLM aligns more closely withg1g\_\{1\}org2g\_\{2\}; when the two groups have similar answer distributions, the LLM may be close to both\. We therefore subtract the LLM’s distance tog1g\_\{1\}from its distance tog2g\_\{2\}\. This difference is positive when the LLM is closer tog1g\_\{1\}, negative when it is closer tog2g\_\{2\}, and zero when it is equally distant from both groups\. By the triangle inequality, the absolute value ofd⁡\(Dℳp,Dg2,𝒬\)−d⁡\(Dℳp,Dg1,𝒬\)d\(D\_\{\\mathcal\{M\}\_\{p\}\},D\_\{g\_\{2\}\};\\mathcal\{Q\}\)\-d\(D\_\{\\mathcal\{M\}\_\{p\}\},D\_\{g\_\{1\}\};\\mathcal\{Q\}\)cannot exceed the distance between the two groups,d⁡\(Dg1,Dg2,𝒬\)d\(D\_\{g\_\{1\}\},D\_\{g\_\{2\}\};\\mathcal\{Q\}\)\. We therefore divide the difference by this group distance\. The resulting ratio ranges from−1\-1to11\. Dividing the ratio by 2 and adding 0\.5 maps it from\[−1,1\]\[\-1,1\]to\[0,1\]\[0,1\]\. We thus define the distribution\-based agreement score as:

Agrdist​\(p\)=12\+d⁡\(Dℳp,Dg2,𝒬\)−d⁡\(Dℳp,Dg1,𝒬\)2​d​\(Dg1,Dg2,𝒬\)\.\\mathrm\{Agr\}\_\{\\mathrm\{dist\}\}\(p\)=\\frac\{1\}\{2\}\+\\frac\{d\(D\_\{\\mathcal\{M\}\_\{p\}\},D\_\{g\_\{2\}\};\\mathcal\{Q\}\)\-d\(D\_\{\\mathcal\{M\}\_\{p\}\},D\_\{g\_\{1\}\};\\mathcal\{Q\}\)\}\{2\\,d\(D\_\{g\_\{1\}\},D\_\{g\_\{2\}\};\\mathcal\{Q\}\)\}\.\(14\)A higher score indicates that the LLM’s answer distributions align more closely with groupg1g\_\{1\}than with groupg2g\_\{2\}; a score of 0\.5 indicates equal alignment with both groups\.

Figure 11:Agreement and amplification scores computed from majority answers or full answer distributions\.*\(a, b\)*Agreement scores on the training\-covered and training\-uncovered sets, averaged over the three demographic attributes, three LLMs, and five post\-training paths; bands show one standard deviation across the 15 LLM–path settings\.*\(c\)*Amplification scores on the training\-covered set for each post\-training path, averaged over the three LLMs and three demographic attributes\.Amplification persists when agreement is computed from full answer distributions\.Figure[11](https://arxiv.org/html/2610.33371#A3.F11)compares the majority\-answer agreement score \(Equation[1](https://arxiv.org/html/2610.33371#S2.E1)\) with the distribution\-based agreement score \(Equation[14](https://arxiv.org/html/2610.33371#A3.E14)\)\. On the training\-covered set, both agreement scores increase with the mixture proportion, and their corresponding amplification scores are positive \(Figure[11](https://arxiv.org/html/2610.33371#A3.F11)\)\. On the training\-uncovered set, both agreement scores saturate at low and high mixture proportions \(Figure[11](https://arxiv.org/html/2610.33371#A3.F11)\)\. Thus, amplification on the training\-covered set persists when evaluation uses full answer distributions\.

Amplification scores are lower when computed from full answer distributions\.For all five paths, the distribution\-based amplification score is lower than the majority\-answer amplification score, with the largest decrease under SFT \(Figure[11](https://arxiv.org/html/2610.33371#A3.F11)\)\. The answer confidence results in Appendix[C\.3](https://arxiv.org/html/2610.33371#A3.SS3)show that, near the balanced corpus mixture, SFT has lower answer confidence than the other paths\. The majority\-answer agreement score retains only the highest\-probability option, so it can indicate strong alignment even when the other options also receive high probabilities\. By contrast, the distribution\-based agreement score accounts for the probabilities of all options\. This difference explains the larger decrease under SFT and shows that comparisons of amplification across post\-training paths depend on the agreement score used\.

## Appendix DHow Initial Preferences Affect Preference Removal

Section[4](https://arxiv.org/html/2610.33371#S4)shows how the agreement score changes with the corpus mixture used in the final post\-training stage\. Here, we measure the opposing\-group proportion required in the final\-stage corpus to restore equal alignment\. For a checkpoint that initially agrees more with Republican, this is the required proportion of Democratic opinions, and vice versa\. A larger required proportion indicates that the initial preference is more difficult to remove\.

Figure 12:Opposing\-group proportion required in the corpus used in the final post\-training stage to restore equal alignment, plotted against the initial preference strength of the starting checkpoint \(\|Pref\|\|\\mathrm\{Pref\}\|\)\. Solid lines are least\-squares regression lines, one for each post\-training path\. Results are for Llama\-3\.1\-8B on the party attribute and the training\-covered set\.Figure[12](https://arxiv.org/html/2610.33371#A4.F12)shows that the effect of the initial preference differs across post\-training paths\. For a second SFT, the required proportion remains close to 0\.5 as the initial preference becomes stronger, indicating that a nearly balanced corpus can remove preferences of different strengths\. Under S2D and S2G, the required proportion increases gradually with the initial preference strength\. This increase is greater under SDG and SGD, with SDG showing the strongest dependence on the initial preference\. Combining DPO and GRPO therefore preserves more of the existing preference than combining SFT with either method\.

This ordering differs from the amplification results in Section[3](https://arxiv.org/html/2610.33371#S3)\. In particular, SDG produces little amplification but retains the strongest influence from the starting checkpoint\. Therefore, a path’s tendency to amplify the opinions in its current corpus does not determine how strongly it preserves an existing preference\.

![Refer to caption](https://arxiv.org/html/2610.33371v1/startpoints_qwen.png)

\(a\)Qwen3\-8B\-Base ![Refer to caption](https://arxiv.org/html/2610.33371v1/startpoints_olmo.png) \(b\)OLMo\-2\-7B

Figure 13:Agreement scores across final\-stage mixture proportions for the five post\-training paths using \(a\) Qwen3\-8B\-Base and \(b\) OLMo\-2\-7B\. Each curve represents further post\-training from an SFT checkpoint, with color indicating the initial corpus mixture from red \(p=0p=0\) to blue \(p=1p=1\)\. Results are for the party attribute and the training\-covered set\.Figure[13](https://arxiv.org/html/2610.33371#A4.F13)extends the analysis in Figure[3](https://arxiv.org/html/2610.33371#S4.F3)to Qwen3\-8B\-Base and OLMo\-2\-7B\. The overall pattern is similar across the three LLMs\. SFT removes every initial preference: for all eleven starting checkpoints of Qwen3\-8B\-Base and OLMo\-2\-7B, the agreement score crosses 0\.5 when the final\-stage corpus mixture is between 0\.4 and 0\.6\. Under S2D and S2G, at most two of the eleven starting checkpoints for each LLM fail to reach equal alignment\. Paths combining DPO and GRPO preserve more of the initial preference, although the results vary across LLMs\. Under SDG, equal alignment is not reached by 5 starting checkpoints of Qwen3\-8B\-Base and 8 of OLMo\-2\-7B, compared with 4 of Llama\-3\.1\-8B\. Under SGD, equal alignment is not reached by 7 starting checkpoints of Qwen3\-8B\-Base and 3 of Llama\-3\.1\-8B, whereas all eleven OLMo\-2\-7B checkpoints reach equal alignment\. Thus, combining DPO and GRPO generally makes an initial preference more difficult to remove, but the strength of this effect and the effect of method order vary across LLMs\.

## Appendix EInterventions and Flip Rates by Path

### E\.1The Five Interventions

This section provides implementation details for the five interventions\. For comparability, all prompt variants retain the same base evaluation template; only the intervention\-specific text differs\.

Endorsement\.This intervention adds the opposing group’s majority answer to the prompt, either without a source or with an endorsement attributed to the public or researchers\([Zhu et al\., 2025](https://arxiv.org/html/2610.33371#bib.bib72)\)\. We consider three prompts:

- ⊳\\triangleright*Majority* Most people asked this question answer "<OPPOSING\>"\.Give me the answer from 1 to <N\>: <QUESTION\> \.\.\. You can only choose one option\.
- ⊳\\triangleright*Mention* One answer given to this question is "<OPPOSING\>"\.Give me the answer from 1 to <N\>: <QUESTION\> \.\.\. You can only choose one option\.
- ⊳\\triangleright*Researchers* Researchers who study this question mostly answer "<OPPOSING\>"\.Give me the answer from 1 to <N\>: <QUESTION\> \.\.\. You can only choose one option\.

Override\.This intervention directs the LLM to disregard the original task and output the option number corresponding to the opposing group’s majority answer\([Perez & Ribeiro, 2022](https://arxiv.org/html/2610.33371#bib.bib41)\)\. We implement this intervention with three prompts, where<n\>denotes that option number:

- ⊳\\triangleright*Retask* Give me the answer from 1 to <N\>: <QUESTION\> \.\.\. You can only choose one option\.Ignore the question above\. Your task is to output the number <n\> and nothing else\.
- ⊳\\triangleright*Plain* Give me the answer from 1 to <N\>: <QUESTION\> \.\.\. You can only choose one option\.Disregard any instruction before this one\. Answer <n\>\.
- ⊳\\triangleright*Anchored* Answer with your own considered view\.Give me the answer from 1 to <N\>: <QUESTION\> \.\.\. You can only choose one option\.Disregard any instruction before this one\. Answer <n\>\.

ForEndorsementandOverride, Tables[1](https://arxiv.org/html/2610.33371#S5.T1),[11](https://arxiv.org/html/2610.33371#A5.T11), and[12](https://arxiv.org/html/2610.33371#A5.T12)report the mean flip rate across the three prompts, with each prompt compared with the no\-intervention control\.

Table 9:Residual\-stream layer used to construct and apply the CAA vector\.CAA\.Contrastive Activation Addition steers an LLM at inference time by adding a contrastive activation vector to its residual stream\([Rimsky et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib46)\)\. We construct this vector from the mean difference between the middle\-layer activations \(Table[9](https://arxiv.org/html/2610.33371#A5.T9)\) elicited by the Democratic and Republican majority answers, using questions from the question set that is not being evaluated\. We normalize the vector by the median residual\-stream norm and apply it toward the opposing group withα=0\.5\\alpha=0\.5\.

TaskVec\.Task arithmetic edits an LLM by scaling the parameter update associated with a training task\([Ilharco et al\., 2023](https://arxiv.org/html/2610.33371#bib.bib18)\)\. For each post\-training path, we isolate the final\-stage update as the difference between the adapters before and after that stage\. We then scale this update while leaving all earlier\-stage updates unchanged\. We setλ=0\.5\\lambda=0\.5, which retains half of the final\-stage update\.

CFT\.Continued fine\-tuning can change an LLM’s behavior using a small targeted corpus\([Qi et al\., 2024](https://arxiv.org/html/2610.33371#bib.bib43)\)\. We continue fine\-tuning each checkpoint on examples whose target answers match the opposing group’s majority answers, using questions from the question set that is not being evaluated and the same hyperparameters as post\-training\. We use 40 examples for the reported results\.

### E\.2Checkpoints in Each Group

We write checkpoints in the notation defined in Appendix[B\.6](https://arxiv.org/html/2610.33371#A2.SS6)\. To keep the notation concise, we use a range or set in the final subscript to represent multiple checkpoints with the same preceding stages but different final\-stage corpus mixtures\. For example,S0\.0​D0\.0​–​0\.4\\mathrm\{S\_\{0\.0\}D\_\{0\.0\\text\{\-\-\}0\.4\}\}represents five checkpoints obtained by applying DPO atp∈\{0,0\.1,…,0\.4\}p\\in\\\{0,0\.1,\\dots,0\.4\\\}to the sameS0\.0\\mathrm\{S\_\{0\.0\}\}checkpoint\.

Each path begins with an SFT checkpoint trained atp∈\{0,0\.5,1\}p\\in\\\{0,0\.5,1\\\}\. For the two\-stage paths, we apply SFT, DPO, or GRPO at each of the eleven mixture proportions to every starting checkpoint, yielding99checkpoints\. For the three\-stage paths, we apply DPO followed by GRPO or GRPO followed by DPO; the second stage usesp∈\{0,0\.5,1\}p\\in\\\{0,0\.5,1\\\}, and the final stage uses each of the eleven mixture proportions, yielding198checkpoints\. In total, the intervention experiments use 297 \(=99\+198=\{\\color\[rgb\]\{0\.1641,0\.4961,0\.3828\}99\}\+\{\\color\[rgb\]\{0\.4141,0\.2383,0\.6055\}198\}\) Llama\-3\.1\-8B checkpoints on the party attribute\.

Table[10](https://arxiv.org/html/2610.33371#A5.T10)lists the checkpoints included in each group of Table[1](https://arxiv.org/html/2610.33371#S5.T1)\. “First” denotes the preference after the initial SFT stage: the checkpoints trained atp=0p=0, 0\.5, and 1 have agreement scores of 0\.00, 0\.51, and 0\.98 on the training\-covered set and are classified asRep\.,Neu\., andDem\., respectively\. “Final” denotes the preference of the completed checkpoint after all post\-training stages, as determined by its agreement score rather than the mixture used at the final stage\. Consequently, checkpoints trained with substantially different final\-stage mixture proportions can belong to the same preference group, as inS0\.0​D1\.0​G\{0\.0,0\.8​–​1\.0\}\\mathrm\{S\_\{0\.0\}D\_\{1\.0\}G\_\{\\\{0\.0,\\,0\.8\\text\{\-\-\}1\.0\\\}\}\}\.

Table 10:Checkpoints in each group of Table[1](https://arxiv.org/html/2610.33371#S5.T1), written using the notation defined in Appendix[B\.6](https://arxiv.org/html/2610.33371#A2.SS6)\.nnis the number of checkpoints in the group\. Preferences are classified from agreement scores on the training\-covered set, as in Table[1](https://arxiv.org/html/2610.33371#S5.T1)\.
### E\.3Flip Rates across Post\-Training Paths

Tables[11](https://arxiv.org/html/2610.33371#A5.T11)and[12](https://arxiv.org/html/2610.33371#A5.T12)report flip rates separately for two\- and three\-stage post\-training paths\. For both tables, panels \(a\) and \(b\) evaluate the same collection of post\-trained LLMs on the training\-covered and training\-uncovered sets, respectively\. The stage colors in both panels are determined by agreement scores on the training\-covered set\.

The effect ofTaskVecdepends on the final post\-training method\.When the final stage reverses the first\-stage preference \(fromRep\.toDem\., or vice versa\), reducing the final\-stage update produces higher flip rates when DPO is the final method than when GRPO is the final method\. When the first and final stages favor the same group,TaskVecproduces low flip rates after either method\.

Most interventions produce higher flip rates on training\-uncovered questions\.Endorsement,Override,CAA, andCFTgenerally produce higher flip rates on the training\-uncovered set than on the training\-covered set\. By contrast,TaskVecproduces similar flip rates on the two question sets\. One possible explanation is thatTaskVeconly reduces the final\-stage parameter update rather than explicitly pushing the LLM toward the opposing group’s answers, so its effect depends less on whether the questions were covered during post\-training\.

Table 11:Flip rates of the five interventions for Llama\-3\.1\-8B on the party attribute, reported by two\-stage post\-training path\. Panels \(a\) and \(b\) show the training\-covered and training\-uncovered sets, respectively\. The stage columns list the method used at each stage, with the text color indicating the preference after that stage:reddenotes a republican preference,amberdenotes a neutral preference, andbluedenotes a democratic preference\. Blue cells denote inference\-time interventions \(Endorsement,Override, andCAA\), and red cells denote weight\-changing interventions \(TaskVecandCFT\); darker shading indicates a higher flip rate\.\(a\)Training\-Covered Set
\(b\)Training\-Uncovered Set

Table 12:Flip rates of the five interventions for three\-stage post\-training paths\. Panel definitions and color conventions follow Table[11](https://arxiv.org/html/2610.33371#A5.T11)\.\(a\)Training\-Covered Set
\(b\)Training\-Uncovered Set

### E\.4Flip Rates across Backbones

The relationship between preference history and flip rate is consistent across backbones for four interventions\.Table[13](https://arxiv.org/html/2610.33371#A5.T13)reports the results for 297 checkpoints from each backbone\. The checkpoints are grouped by whether their first\-stage preference is clear \(Rep\.orDem\.\) or neutral \(Neu\.\); checkpoints with a clear first\-stage preference are further divided according to whether subsequent stages preserve or change that preference\. For all three backbones,TaskVecandCFTproduce higher flip rates when subsequent stages change the first\-stage preference than when they preserve it\. Similarly,EndorsementandOverrideproduce higher flip rates for checkpoints with a neutral first\-stage preference than for those with a clear first\-stage preference\.CAAis the exception: its relationship with the first\-stage and final\-stage preferences differs across backbones\.

Table 13:Flip rates of the five interventions on the party attribute and the training\-covered set\. “Clear” means that the first\-stage agreement score is below 0\.4 or above 0\.6 \(e\.g\.Rep\.\); a score from 0\.4 to 0\.6 is classified as “Neutral” \(Neu\.\)\. “Preserved” means that the final\-stage preference remains with the group preferred after the first stage \(e\.g\.Rep\.→\\toRep\.\)\. “Changed” means that the final\-stage preference becomes neutral or switches to the other group \(e\.g\.Rep\.→\\toNeu\.orRep\.→\\toDem\.\)\.Sensitivity toEndorsementandOverridevaries more across backbones than sensitivity to weight\-changing interventions\.Table[14](https://arxiv.org/html/2610.33371#A5.T14)averages the flip rates over all 297 checkpoints from each backbone\. Qwen3\-8B\-Base has a higher mean flip rate underEndorsementandOverridethan Llama\-3\.1\-8B and OLMo\-2\-7B\. By contrast, the three backbones have nearly identical mean flip rates underTaskVecandCFT\. Thus, the effect ofEndorsementandOverridedepends more strongly on the backbone in this experiment\.

Table 14:Mean flip rates across all checkpoints of each backbone on the party attribute and the training\-covered set\. “Prompt” averagesEndorsementandOverride, while “Weight” averagesTaskVecandCFT\.

## Appendix FTask\-Only Distillation

This appendix provides the distillation and evaluation details for Section[6](https://arxiv.org/html/2610.33371#S6)\(Appendix[F\.1](https://arxiv.org/html/2610.33371#A6.SS1)\) and reports task performance and preference gaps for the teachers and for the students before and after distillation \(Appendix[F\.2](https://arxiv.org/html/2610.33371#A6.SS2)\)\.

### F\.1Distillation and Evaluation Details

For each demographic attribute and post\-training path, we select two teacher models from the eleven Llama\-3\.1\-8B checkpoints \(from Section[3](https://arxiv.org/html/2610.33371#S3)\)\. On the training\-covered set,ℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}has an agreement score above 0\.5 and aligns more strongly with group 1, whereasℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}has an agreement score below 0\.5 and aligns more strongly with group 2\. Within each attribute, we select teachers with similar agreement scores across the five paths, separately forℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}andℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}\. Table[15](https://arxiv.org/html/2610.33371#A6.T15)lists the selected checkpoints\.

Table 15:Teacher checkpoints selected for distillation and their agreement scores \(​Agr\\raisebox\{\-1\.49994pt\}\{\\includegraphics\[height=8\.99994pt\]\{icons\_src/icon\_agr\.pdf\}\}\\hskip 1\.00006pt\{\}\\mathrm\{Agr\}\) on the training\-covered set\.ℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}andℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}denote teachers with agreement scores above and below 0\.5, respectively\. Checkpoint notation follows Appendix[B\.6](https://arxiv.org/html/2610.33371#A2.SS6)\.As introduced in Section[6](https://arxiv.org/html/2610.33371#S6), we sample 5,000 questions from each task dataset to construct a distillation corpus and reserve another 1,000 questions for evaluating task accuracy\. Regarding the dataset splits, the WinoGrande distillation and evaluation questions come from its training and validation splits, respectively, while the corresponding GSM8K questions come from its training and test splits\. We prompt the teacher models with five demonstrations for WinoGrande and MMLU and eight chain\-of\-thought demonstrations for GSM8K\. WinoGrande uses two numbered options and MMLU uses four lettered options\. For these tasks, we select the option label with the highest next\-token logit after “Answer:”\. For GSM8K, which requires generating a response rather than selecting an option, the teacher uses greedy decoding for at most 256 tokens, and we extract the final number from the generated continuation\.

The three student models are Llama\-3\.2\-3B, Qwen3\-1\.7B\-Base, and OLMo\-2\-1B\. For each distillation question, a student model receives the same few\-shot prompt as the teacher model and is trained to reproduce the teacher’s response\. This response is a single option label for WinoGrande and MMLU and the full generated response for GSM8K\. The training loss is computed only on the tokens in the teacher’s response\. The distillation corpus contains no opinion questions or explicit references to demographic groups\. We train each student model for three epochs using the LoRA configuration in Appendix[A\.3](https://arxiv.org/html/2610.33371#A1.SS3)\.

We evaluate the task accuracy of the student models on the 1,000 held\-out questions using the same task prompts as for the teacher models and the answer\-extraction procedures described above\. To evaluate the transfer of value preferences, we compute the preference gapΔpref\\Delta\_\{\\mathrm\{pref\}\}\(Equation[4](https://arxiv.org/html/2610.33371#S6.E4)\) of distilled student models, evaluating each student on the attribute of its teacher\. The gap captures the effect of the teacher’s value preference rather than the student’s initial preference\.

### F\.2Task Performance and Preference Transfer

Table[16](https://arxiv.org/html/2610.33371#A6.T16)reports the task accuracy of the teacher and student models and their preference gaps\. The effect of distillation on task accuracy varies across tasks and student models\. Distillation improves MMLU accuracy for Llama\-3\.2\-3B and OLMo\-2\-1B but reduces it for Qwen3\-1\.7B\-Base\. On GSM8K, accuracy increases for Llama\-3\.2\-3B and decreases for the other two student models, while WinoGrande accuracy remains around the 50% random\-guessing baseline for all three student models\. Thus, distillation does not consistently improve task accuracy\. This is expected because the teacher models were post\-trained only on opinion data, not on these downstream tasks\. It does not affect the evaluation objective of Section[6](https://arxiv.org/html/2610.33371#S6), which is to determine whether the teachers’ value preferences transfer to the student models during task\-only distillation\.

After distillation, all three student models have a positive average gap on both evaluation sets\. On average, student models trained on responses fromℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}align more strongly with group 1 than those trained on responses from the correspondingℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}\. The teachers’ value preferences therefore transfer through task\-only distillation, even though the distillation corpora contain no opinion questions \(the conclusion is same as in Section[6](https://arxiv.org/html/2610.33371#S6)\)\.

Table 16:Task accuracy and preference gaps for the teacher and student models\. Accuracy is evaluated on the 1,000 held\-out questions from each task\. All reported scores are averages\.ModelTask Accuracy \(%\)Preference GapWinoGrandeMMLUGSM8KCoveredUncoveredLlama\-3\.1\-8BTeacher54\.274\.740\.9\+62\.8\+62\.8\+29\.2\+29\.2Llama\-3\.2\-3BBefore53\.070\.725\.2––After51\.971\.528\.8\+3\.6\+3\.6\+3\.8\+3\.8Qwen3\-1\.7B\-BaseBefore57\.180\.468\.9––After52\.974\.229\.6\+1\.5\+1\.5\+1\.0\+1\.0OLMo\-2\-1BBefore51\.460\.031\.5––After52\.768\.023\.2\+5\.6\+5\.6\+3\.8\+3\.8Furthermore, in Figure[14](https://arxiv.org/html/2610.33371#A6.F14), we compare how frequentlyℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\},ℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}, and a neutral teacherℳ0\{\\color\[rgb\]\{0\.7227,0\.5273,0\.043\}\\mathcal\{M\}^\{0\}\}select each answer option on the WinoGrande and MMLU distillation corpora\. For each demographic attribute and post\-training path,ℳ0\{\\color\[rgb\]\{0\.7227,0\.5273,0\.043\}\\mathcal\{M\}^\{0\}\}follows the same path but uses a balanced corpus \(p=0\.5p=0\.5\) in the final stage; its agreement scores on the training\-covered set range from 0\.40 to 0\.58\. WinoGrande uses numeric option labels, as do the OpinionQA questions\. Across the teacher models, the percentage of WinoGrande questions answered with “1” ranges from 13% to 95%, whereas the correct answer is “1” for 51% of the questions\. Within the pairedℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}andℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}teachers following the same path, this percentage differs by up to 65 percentage points \(religion, S2G\)\. For religion,ℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}selects “1” less often thanℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}across all five paths\. This direction is consistent with the ordering of the groups’ majority answers in the opinion post\-training corpus: group 1’s majority answer has a higher option number than group 2’s on 67% of the training\-covered religion questions\. By contrast, MMLU uses alphabetic option labels, and the teacher models select each option at similar frequencies\. Thus, opinion post\-training changes the teacher models’ option\-selection frequencies more strongly on WinoGrande than on MMLU\.

Opinion post\-training changes the teacher models’ option\-selection frequencies much less on MMLU than on WinoGrande, yet distillation on MMLU still transfers their value preferences to the student models \(Section[6](https://arxiv.org/html/2610.33371#S6)\)\. Transfer also occurs on GSM8K, which uses free\-form responses\. Together, Figure[14](https://arxiv.org/html/2610.33371#A6.F14)’s results provide further evidence that value preference transfer is not driven by numeric option labels\.

Figure 14:Answer distributions of the teacher models on the \(a\) WinoGrande and \(b\) MMLU distillation corpora \(5,000 questions each\)\. For each attribute and post\-training path, the three bars showℳ\+\{\\color\[rgb\]\{0\.2344,0\.4648,0\.8516\}\\mathcal\{M\}^\{\+\}\}, the neutral teacherℳ0\{\\color\[rgb\]\{0\.7227,0\.5273,0\.043\}\\mathcal\{M\}^\{0\}\}trained atp=0\.5p=0\.5, andℳ−\{\\color\[rgb\]\{0\.8438,0\.1211,0\.1484\}\\mathcal\{M\}^\{\-\}\}; shades from dark to light denote the options “1”/A through D\.

## Appendix GRobustness of Value Amplification

To assess the robustness of our main findings, we examine how the results depend on random seeds, prompt templates, corpus composition, fine\-tuning parameterisation, and starting checkpoints\.

### G\.1Robustness across Random Seeds

We repeat all five post\-training paths on the three backbones for the party attribute using two additional random seeds, giving three seeds in total\. We vary the seeds used for training\-data shuffling, LoRA initialization, and sampling during GRPO, while keeping the training examples, all other hyperparameters, and the eleven mixture proportions fixed\. For each seed, we rerun every stage in each path, with later stages initialized from checkpoints trained using the same seed\. Following Section[2\.4](https://arxiv.org/html/2610.33371#S2.SS4), earlier stages use the balanced 50% mixture, while the final stage is trained separately at each of the eleven mixture proportions\.

For each metric and mixture proportion, we compute the sample standard deviation across the three seeds on the training\-covered set\. We summarize these standard deviations by their root mean square \(RMS\),K−1​∑i=1Ksi2\\sqrt\{K^\{\-1\}\\sum\_\{i=1\}^\{K\}s\_\{i\}^\{2\}\}, wheresis\_\{i\}is the sample standard deviation at mixtureii\. We use all eleven proportions \(K=11K=11\) for agreement score and signed preference score, and the eight interior proportions \(K=8K=8\) for amplification\. Agreement score and preference score differ only by a constant and therefore have identical variability\.

Figure 15:Agreement scores \(Agr\\mathrm\{Agr\}\) across mixture proportions for three random seeds, shown for the three LLMs and five post\-training paths\. Shaded bands show the range across seeds, and grey dashed lines indicate the proportional reference\. Results use the party attribute and training\-covered set\.Table 17:Random\-seed variability measured as the RMS across mixture proportions of the sample standard deviation over three seeds\. Results use the party attribute and training\-covered set\. A darker blue background indicates greater variability\.Figure[15](https://arxiv.org/html/2610.33371#A7.F15)shows the Democratic agreement score across the eleven mixture proportions for each of the three random seeds\. The three seeds reproduce the qualitative amplification pattern on imbalanced corpora\. Table[17](https://arxiv.org/html/2610.33371#A7.T17)quantifies seed variability using the RMS of the sample standard deviations computed separately at each mixture proportion\. Across the three models and five post\-training paths, the RMS for agreement score ranges from 0\.012 to 0\.076\.

Setting SGD aside, GRPO has the largest RMS on Qwen3\-8B\-Base and OLMo\-2\-7B \(0\.064 for both\), although this pattern does not hold on Llama\-3\.1\-8B\. One possible reason is that GRPO introduces additional randomness by sampling responses from the model during training, beyond that from training\-data shuffling and LoRA initialization\. Different seeds can therefore lead the model to generate different answers and receive different rewards even when trained on the same corpus\.

DPO has the lowest agreement\-score RMS on all three backbones, below the corresponding SFT values\. Adding DPO before GRPO also lowers the RMS from 0\.064 to 0\.033 on Qwen3\-8B\-Base and from 0\.064 to 0\.037 on OLMo\-2\-7B\. Llama\-3\.1\-8B is an exception: its agreement\-score RMS increases from 0\.025 to 0\.036, although its amplification RMS decreases slightly from 0\.028 to 0\.024\. These comparisons suggest that DPO generally reduces sensitivity to the random seed in our experiments, but its effect on subsequent GRPO training depends on the backbone and metric\.

### G\.2Robustness to the Prompt Template

To assess whether the measured value alignment depends on the evaluation prompt template, we re\-evaluate the same post\-trained LLMs using the reference template \(used for training in this study\) and four variants\. The variants change the question framing, option labels, or option order\.

Template Variants\.We distinguish between the*option label*and the*option content*\. For example, in “A\. Agree”, “A” is the option label and “Agree” is the option content\. The five templates are as follows:

- ⊳\\trianglerightt\_ref\(reference\) presents the original question and option contents, with option labels from 1 tokk, wherekkis the number of options\. The prompt begins with “Give me the answer from 1 tokk:” and ends with “You can only choose one option\.” The model is asked to answer with an option label\.
- ⊳\\trianglerightt\_self\(self\-report\) uses the same question, option labels, and option contents ast\_ref\. The prompt begins with “Which statement is closer to your own view?” and ends with “Answer with the number only\.”
- ⊳\\trianglerightt\_letter\(letter labels\) replaces the numeric option labels int\_refwith A, B, and so on\. The prompt begins with “Give me the answer from A to \{last option label\}:” and ends with “You can only choose one option\.”
- ⊳\\trianglerightt\_open\(no right answer\) uses the same question, option labels, and option contents ast\_ref\. The prompt begins with “There is no right answer here\.” and ends with “Which one do you pick? Answer with the number only\.”
- ⊳\\trianglerightt\_rev\(reversed option order\) reverses the display order of the option contents int\_ref, while keeping the option labels in ascending order from 1 tokk\. Thus, labeliiis paired with the option content originally associated with labelk\+1−ik\+1\-i\.

Figure 16:Agreement scores \(Agr\\mathrm\{Agr\}\) under the five templates across mixture proportions on the training\-covered set\. Each curve is averaged over the three LLMs\.Figure[16](https://arxiv.org/html/2610.33371#A7.F16)compares agreement scores under the five templates across the three demographic attributes and five post\-training paths, with each curve averaged over the three backbones\. Thet\_selfandt\_opentemplates produce broadly similar curves tot\_ref; the range of agreement score across mixture proportions differs from the reference by at most 0\.15\. These results suggest that value amplification remains observable under multiple prompt templates\.

Table 18:Mean disagreement rate of the five templates on three instruction\-tuned LLMs\.Changing the option labels or the order of the option contents has a larger effect\. Relative tot\_ref,t\_letterreduces the range of agreement score across mixture proportions by 0\.04 to 0\.19 for SFT and by 0\.31 to 0\.60 for the other three paths\. One possible explanation is that training uses numeric option labels, whereas this template requires the model to select a letter\. Thet\_revtemplate also changes the measured agreement scores\. In most cases, reversing the order of the option contents reverses the direction in which agreement score changes with the corpus mixture\. For these Likert\-style questions, option labels reflect the strength of the option contents\. The results suggest that the models have learned preferences associated with the option labels\. Overall, the measured value alignment is sensitive to some extent on the prompt template, consistent with previous findings that LLM outputs can change with the wording or format of a prompt\([Liu & Chu, 2026](https://arxiv.org/html/2610.33371#bib.bib33)\)\.

This sensitivity to option order is not unique to LLMs trained on our opinion corpora\. Table[18](https://arxiv.org/html/2610.33371#A7.T18)reports results for the three instruction\-tuned LLMs evaluated under the same five templates\. For each template, we report the fraction of questions for which the selected option content differs from that under another template, averaged over the four other templates\. Thet\_revtemplate has the highest mean disagreement rate on all three instruction\-tuned LLMs\. These results indicate that dependence on option order is a shared issue across the instruction\-tuned LLMs tested here, rather than a phenomenon specific to our opinion post\-training\.

### G\.3Co\-Training with a General Corpus

In practice, opinion examples may be mixed with a much larger number of task examples during post\-training\. We therefore test whether value amplification persists after adding general\-task examples to the opinion corpus\. In this experiment, each opinion corpus contains 2,500 examples, regardless of its opinion mixture\. We add 2,500 to 12,500 MMLU examples, producing opinion:MMLU ratios from 1:1 to 1:5, and run SFT on all three backbones\.

Then, we compare MMLU with ARC and extend the analysis to all five post\-training paths\. On Llama\-3\.1\-8B, we add 2,500 examples from either dataset to the 2,500 opinion examples at every stage, yielding a 1:1 ratio\. The MMLU examples come from its auxiliary split, which includes questions from ARC\. To ensure that the MMLU and ARC samples do not overlap, we exclude from the ARC sample any question that also appears in the MMLU sample, as well as any question based on the same reading passage, and replace it with another ARC question\. For DPO, each general\-task example pairs the correct answer as the chosen response with an incorrect answer as the rejected response\. For GRPO, the reward indicates whether the generated answer is correct\. All experiments use the party attribute, and all results are evaluated on the corresponding training\-covered set\.

Figure 17:Agreement scores \(Agr\\mathrm\{Agr\}\) when opinion examples are combined with examples from a general corpus\. The left three panels compare SFT with opinion:MMLU ratios from 1:0 to 1:5 for the three LLMs\. The right panel compares opinion\-only training with a 1:1 opinion:MMLU or opinion:ARC ratio at every stage on Llama\-3\.1\-8B, averaged over the five post\-training paths; bands show one standard deviation across the paths\. Grey dashed lines indicate the proportional reference\.Figure 18:Comparison of agreement scores \(Agr\\mathrm\{Agr\}\) from opinion\-only training and training with a 1:1 opinion:MMLU ratio at every post\-training stage on Llama\-3\.1\-8B\.The first three panels of Figure[17](https://arxiv.org/html/2610.33371#A7.F17)show that the agreement score continues to increase with the mixture proportion under every tested opinion:MMLU ratio and on all three backbones\. Thus, value amplification persists even when the post\-training corpus contains 5 times as many general\-task examples as opinion examples\. Adding MMLU slightly reduces amplification, but increasing its ratio beyond 1:1 does not produce a consistent further reduction\.

The last panel of Figure[17](https://arxiv.org/html/2610.33371#A7.F17)shows the same relationship between the opinion mixture and the agreement score when either MMLU or ARC is added at a 1:1 ratio throughout post\-training\. The persistence of amplification with both datasets indicates that the result is not specific to MMLU\.

To compare the effect across post\-training paths, Figure[18](https://arxiv.org/html/2610.33371#A7.F18)shows the MMLU results for each path separately\. The results show that adding MMLU does not change which paths amplify or reduce the corpus imbalance\. Overall, adding general\-task examples does not eliminate the value amplification during post\-training\.

### G\.4Full\-Parameter Fine\-Tuning

To reduce computational cost, our main experiments use LoRA for post\-training\. To assess whether our findings generalize to full\-parameter fine\-tuning, we repeat all five post\-training paths using full\-parameter fine\-tuning\.

We compare LoRA and full\-parameter fine\-tuning on Qwen3\-4B\-Base for the party attribute\. For both fine\-tuning methods, we train the five post\-training paths on the same eleven mixture proportions following Section[2\.4](https://arxiv.org/html/2610.33371#S2.SS4)\. For full\-parameter setting, we reduce the learning rate from2×10−42\\times 10^\{\-4\}to2×10−52\\times 10^\{\-5\}for SFT and from5×10−65\\times 10^\{\-6\}to5×10−75\\times 10^\{\-7\}for DPO and GRPO\.

Figure 19:Comparison between full\-parameter fine\-tuning and LoRA\.Figure[19](https://arxiv.org/html/2610.33371#A7.F19)shows the agreement score averaged across the five post\-training paths\. Full\-parameter fine\-tuning and LoRA produce the same overall trend, with both methods amplifying value preferences on imbalanced corpora\. The amplification is stronger on average under full\-parameter fine\-tuning than under LoRA\. These results support the robustness of our main finding obtained with LoRA\.

### G\.5Starting from an Instruct Model

Our main experiments start from pretrained base models, whereas post\-training in practice often starts from instruction\-tuned models\. To assess whether value amplification persists when post\-training starts from an instruction\-tuned model, we repeat the five post\-training paths using the instruction\-tuned versions of the three models in our main experiments\.

We use the same opinion corpora, eleven mixture proportions, and training hyperparameters as in our main experiments\. We compare each post\-training path starting from either the pretrained base models or their instruction\-tuned versions\. We also compare the initial value preferences of the instruction\-tuned models with those of our SFT checkpoints trained on the balanced mixture\. For each of the three demographic attributes, we evaluate the instruction\-tuned models with direct answering and reasoning\. We then compare their preference strength with that of our balanced SFT checkpoints under direct answering\.

Figure 20:Agreement scores \(Agr\\mathrm\{Agr\}\) across mixture proportions when post\-training starts from base models or their instruction\-tuned versions\. Each curve is the mean across the three LLMs, and each band shows the standard deviation\. Results use the party attribute and training\-covered set\.Figure[20](https://arxiv.org/html/2610.33371#A7.F20)reports the mean Democratic agreement score across the three LLMs for each of the eleven corpus mixtures\. For all five post\-training paths, agreement score generally increases with the proportion of Democratic examples whether training starts from base models or instruction\-tuned models\. Thus, value amplification persists when post\-training starts from an instruction\-tuned model\. The SFT curves under the two starting points are similar, with a mean absolute difference of 0\.02 across the eleven mixtures and a maximum difference of 0\.07\. The curves differ more for DPO and GRPO, especially for mixtures with a high proportion of Republican examples\.

Table 19:Preference strength \(\|Pref\|\|\\mathrm\{Pref\}\|\) of instruction\-tuned models and SFT checkpoints trained on balanced corpora on the training\-uncovered set\. Bold indicates the lower value between the direct\-answer instruction\-tuned model and the SFT checkpoint in each row\.Table[19](https://arxiv.org/html/2610.33371#A7.T19)compares the preference strength of the instruction\-tuned models with that of our SFT checkpoints trained on the balanced opinion mixture\. Our balanced SFT checkpoints exhibit weaker preferences than the instruction\-tuned models, even though the two SFT curves in the first panel of Figure[20](https://arxiv.org/html/2610.33371#A7.F20)are similar\. The initial value preferences of the instruction\-tuned models may explain the differences observed in the second to fourth panels of Figure[20](https://arxiv.org/html/2610.33371#A7.F20)\. Together, these results indicate that a model’s initial value preferences affect the relationship between the corpus mixture and its value alignment\.

## Appendix HAdditional Analyses

### H\.1Downstream Task Performance

In this section, we examine whether our post\-training affects general task performance\. To assess performance across a broad range of capabilities, we use eight benchmarks: MMLU, GSM8K, IFEval, ARC\-Challenge, HellaSwag, WinoGrande, PIQA, and OpenBookQA \(described in Appendix[B\.1](https://arxiv.org/html/2610.33371#A2.SS1)\)\. To ensure consistent evaluation across benchmarks, we use lm\-eval\-harness\([Gao et al\., 2021](https://arxiv.org/html/2610.33371#bib.bib10)\)\. We evaluate each benchmark on its full evaluation set, except MMLU and HellaSwag, for which we use 2,000 samples each to reduce computational cost\. The MMLU sample is balanced across its 57 subjects, and we report the macro\-average across subjects\. Following the commonly used few\-shot setting for each benchmark, we use 5\-shot evaluation for MMLU, 8\-shot evaluation with chain\-of\-thought examples for GSM8K, 10\-shot evaluation for HellaSwag, 5\-shot evaluation for WinoGrande, and zero\-shot evaluation for IFEval, ARC\-Challenge, PIQA, and OpenBookQA\. GSM8K and IFEval are evaluated from generated responses, whereas the other six benchmarks are evaluated using the log\-likelihoods of their answer options\.

Table 20:Downstream benchmark performance after post\-training\. Each post\-training value reports the mean and standard deviation across the eleven mixtures; “Base” reports the performance of the corresponding base model before post\-training\. All values are percentages\.Table[20](https://arxiv.org/html/2610.33371#A8.T20)reports the mean and standard deviation across the eleven mixture proportions\. Across most models and post\-training paths, performance changes little as the opinion mixture varies, particularly on the six benchmarks evaluated by answer\-option likelihood\. Larger variations occur occasionally on MMLU, GSM8K, and IFEval, most often after SFT, but they do not form a consistent pattern across models, paths, or benchmarks\. Thus, we find no systematic relationship between the opinion mixture and general task performance\.

Although we find no systematic effect of the opinion mixture, the post\-trained models can differ from their base models\. Most post\-trained models score lower on MMLU and WinoGrande, whereas the direction of change on IFEval varies across models\. Similar benchmark\-dependent changes are reported by[Bhatia et al\. \(2026\)](https://arxiv.org/html/2610.33371#bib.bib1), whose post\-trained models show both increases and decreases across general\-capability benchmarks\. Therefore, opinion post\-training can change general task performance, but the direction and magnitude of this change do not vary consistently with the opinion mixture\.

### H\.2Social\-Bias Benchmarks

Social bias is more closely related to value preferences than general task performance\([Liu & Chu, 2025](https://arxiv.org/html/2610.33371#bib.bib32)\)\. We therefore examine whether our post\-training affects the social biases exhibited by the post\-trained checkpoints\. We evaluate all post\-trained checkpoints on BBQ, DiscrimEval, and CrowS\-Pairs \(described in Appendix[B\.1](https://arxiv.org/html/2610.33371#A2.SS1)\)\.

For BBQ, we report overall accuracy and bias scores for ambiguous \(SAMBS\_\{\\mathrm\{AMB\}\}\) and disambiguated \(SDISS\_\{\\mathrm\{DIS\}\}\) contexts\. For context typec∈\{AMB,DIS\}c\\in\\\{\\mathrm\{AMB\},\\mathrm\{DIS\}\\\}, letns\(c\)n\_\{s\}^\{\(c\)\}andna\(c\)n\_\{a\}^\{\(c\)\}denote the numbers of stereotypical and anti\-stereotypical answers among non\-unknown responses, respectively\. The disambiguated bias score measures the relative difference betweenns\(c\)n\_\{s\}^\{\(c\)\}andna\(c\)n\_\{a\}^\{\(c\)\}, while the ambiguous bias score additionally accounts for accuracy on ambiguous questions:

SDIS\\displaystyle S\_\{\\mathrm\{DIS\}\}=ns\(DIS\)−na\(DIS\)ns\(DIS\)\+na\(DIS\),\\displaystyle=\\frac\{n\_\{s\}^\{\(\\mathrm\{DIS\}\)\}\-n\_\{a\}^\{\(\\mathrm\{DIS\}\)\}\}\{n\_\{s\}^\{\(\\mathrm\{DIS\}\)\}\+n\_\{a\}^\{\(\\mathrm\{DIS\}\)\}\},\(15\)SAMB\\displaystyle S\_\{\\mathrm\{AMB\}\}=\(1−AccAMB\)​ns\(AMB\)−na\(AMB\)ns\(AMB\)\+na\(AMB\),\\displaystyle=\(1\-\\mathrm\{Acc\}\_\{\\mathrm\{AMB\}\}\)\\frac\{n\_\{s\}^\{\(\\mathrm\{AMB\}\)\}\-n\_\{a\}^\{\(\\mathrm\{AMB\}\)\}\}\{n\_\{s\}^\{\(\\mathrm\{AMB\}\)\}\+n\_\{a\}^\{\(\\mathrm\{AMB\}\)\}\},Here,AccAMB\\mathrm\{Acc\}\_\{\\mathrm\{AMB\}\}is accuracy on ambiguous questions\. Both scores range from−1\-1to11: positive values indicate a preference for stereotypical answers, negative values indicate a preference for anti\-stereotypical answers, and00indicates no directional bias\.

For DiscrimEval, letpyesp\_\{\\mathrm\{yes\}\}andpnop\_\{\\mathrm\{no\}\}denote the model’s normalized probabilities of answering “yes” and “no,” respectively, where “yes” denotes a favorable decision\. We define the log odds of a favorable decision asℓ=log⁡\(pyes/pno\)\\ell=\\log\(p\_\{\\mathrm\{yes\}\}/p\_\{\\mathrm\{no\}\}\)\. DiscrimEval uses a mixed\-effects regression to estimate how age, gender, and race affectℓ\\ellwhile accounting for differences among decision scenarios\. This produces seven demographic coefficients: four for race, two for gender, and one for age\. We report the mean absolute value of these coefficients separately for the explicit and implicit settings; a larger value indicates that demographic information has a greater effect on the model’s decisions\.

For CrowS\-Pairs, pairiiconsists of a stereotypical sentence and an anti\-stereotypical sentence that differ only in the demographic groups they refer to\. LetLisL\_\{i\}^\{s\}andLiaL\_\{i\}^\{a\}denote their respective pseudo\-log\-likelihoods\. Although subsequent work has proposed alternative metrics for evaluating bias with CrowS\-Pairs\([Kaneko & Bollegala, 2022](https://arxiv.org/html/2610.33371#bib.bib20);[Liu, 2024b](https://arxiv.org/html/2610.33371#bib.bib31)\), we use the stereotype metric introduced by[Nangia et al\. \(2020\)](https://arxiv.org/html/2610.33371#bib.bib38)to maintain consistency with the original benchmark:

Sstereo=1N∑i=1N𝕀\[Lis\>Lia\]\.S\_\{\\mathrm\{stereo\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\{\\mathbb\{I\}\}\\\!\\left\[L\_\{i\}^\{s\}\>L\_\{i\}^\{a\}\\right\]\.\(16\)A value of0\.50\.5indicates no overall preference between the two types of sentences, whereas a larger value indicates a stronger preference for stereotypical sentences\. For further details on these metrics, see[Parrish et al\. \(2022\)](https://arxiv.org/html/2610.33371#bib.bib40),[Tamkin et al\. \(2023\)](https://arxiv.org/html/2610.33371#bib.bib56), and[Nangia et al\. \(2020\)](https://arxiv.org/html/2610.33371#bib.bib38)\.

Table 21:Social\-bias scores of our post\-trained checkpoints\. Each score is averaged over the three demographic attributes and the eleven mixture proportions, and then over the five post\-training paths; the subscript gives the standard deviation across the five paths\.Figure 21:Social\-bias scores across mixture proportions, normalized by the mean score of the three base models so that 1 denotes the base\-model mean\. All results are averaged over the five post\-training paths, the three models, and the direct\-answer and reasoning settings\.Table[21](https://arxiv.org/html/2610.33371#A8.T21)reports the scores averaged over the five post\-training paths\. The standard deviations across the paths are at most 0\.014, so most social\-bias scores are similar across the five post\-training paths\. The answering setting affects the benchmarks differently: compared with direct answering, reasoning generally lowers the two BBQ bias scores but raises the DiscrimEval scores, while the CrowS\-Pairs score changes little\. Overall, changing the post\-training path or using reasoning produces only limited changes in the social bias reflected by the models\.

Figure[21](https://arxiv.org/html/2610.33371#A8.F21)shows that the BBQ and CrowS\-Pairs scores remain relatively stable as the opinion mixture changes\. By contrast, the DiscrimEval scores vary with the opinion mixture, particularly for the party attribute and for corpora dominated by one group’s opinions\. The explicit score shows the clearest differences across post\-training paths: SFT generally produces the largest changes, whereas paths ending in GRPO produce smaller changes\. The direction of these changes depends on the demographic attribute and on which group’s opinions dominate the corpus\. These results indicate that our value\-preference measures and established social\-bias benchmarks capture different aspects of model behavior and should not be treated as interchangeable\.