Where does output diversity collapse in post-training?

# Where does output diversity collapse in post-training?

Source: https://arxiv.org/html/2604.16027

Constantinos Karouzos
Xingwei Tan
Nikolaos Aletras

School of Computer Science
University of Sheffield, UK
{kkarouzos1, xingwei.tan, n.aletras}@sheffield.ac.uk

## Abstract

Post-trained language models produce less varied outputs than their base counterparts. This output diversity collapse undermines inference-time scaling methods that rely on varied samples, and risks homogenizing model outputs on creative and value-laden tasks. Prior work attributes collapse to specific post-training methods, without separating the role of training data composition from the method, or the generation format from the model weights. We trace output diversity through three parallel post-training lineages of Olmo 3, Think (chain-of-thought distillation), Instruct (broad multi-source data), and RL-Zero, across 15 tasks and four text diversity metrics. We find that the location of collapse co-varies with data composition: the Think lineage loses most semantic diversity at supervised fine-tuning, and the effect of DPO is larger in Instruct than in Think. Suppressing chain-of-thought reasoning at inference in Think models drops accuracy on hard tasks, yet leaves answer-level diversity unchanged, showing that the collapse is embedded in the model weights by training data, not imposed by the generation format. Decomposing diversity loss on six verifiable tasks into a quality-control component (removal of incorrect outputs) and a residual component (genuine narrowing among correct outputs) reveals that the split is task-dependent, and Think models retain more correct-answer diversity than Instruct despite collapsing more in aggregate. Our results indicate that diversity collapse is determined during training by data composition and cannot be addressed at inference time alone.

Code: https://github.com/ckarouzos/where-diversity-collapses/

## 1 Introduction

Large language models (LLMs) rely on post-training to improve helpfulness, safety, and instruction compliance. Post-training combines supervised fine-tuning (SFT) on curated demonstrations, and direct preference optimization (DPO) or reinforcement learning from human feedback (RLHF). However, this results in output diversity collapse, i.e., models produce more uniform outputs than their base counterparts across summarization, reasoning, and open-ended generation. Diversity collapse limits self-consistency, pass@k sampling, and test-time compute scaling. Recent work shows diversity is the mechanism underlying inference scaling laws. The algorithmic causes are well-understood, yet diversity collapses across task types. This leads LLMs to produce less diverse outputs than a basic web search, co-writing with LLMs reduces content diversity, and single-reward RLHF can amplify majority preferences to near-total dominance. Yet, prior work attributes collapse to specific algorithms: DPO in narrative generation, the reward step in creative tasks, and SFT in reasoning, without investigating the effect of *data* compositions. Recent work suppresses chain-of-thought (CoT) at inference but measures only accuracy, not diversity. No existing study isolates the role of the training *method* from the training *data*, or the generation *format* from the model weights.

Two questions remain open: (1) does the diversity collapse co-vary with the post-training method or with the post-training data composition, and (2) does the CoT format itself constrain diversity at inference, or is the collapse embedded in the model weights?

We answer these questions through a controlled experimental setting (Figure 1). We monitor the output diversity of the open weight and data Olmo 3 model family, which releases checkpoints of all post-training stages across three parallel lines. Think and Instruct variants share the same post-training recipe (SFT → DPO → RL) but differ in data, while RL-Zero bypasses SFT and DPO entirely. Evaluating 13 models across 15 tasks with four diversity metrics, we show that the same post-training method produces different diversity outcomes depending on the upstream data composition, and that each stage plays a distinct role.

Our contributions:

- We compare Think vs. Instruct lineages, showing that collapse location depends on data: narrow CoT distillation for Think models is associated with a larger drop at SFT, while the DPO drop is larger in Instruct models (§4.1);
- We evaluate Think models with CoT suppressed at inference and find no diversity recovery on any task–stage combinations, while quality drops. Diversity collapse resides in the model weights, not in the CoT generation format (§4.2);
- We decompose diversity reduction into a quality-control component (removal of incorrect outputs) and a residual component (genuine narrowing among correct outputs), showing the split is task-dependent (§4.3).

## 2 Related Work

### The reliability–diversity tradeoff in post-training

Recent work shows that aligned models exhibit high output homogeneity across a wide range of model families and scales. RLHF reduces both per-input and across-input diversity. Human co-writing with aligned models reduces content diversity, and users brainstorming with ChatGPT produce less semantically distinct ideas. In reasoning, SFT improves pass@1 but degrades pass@k; base models outperform RLVR-trained models at large sample budgets, and base models produce more diverse outputs. Some work identified DPO as the steepest drop. Recent work shows that under domain shift the adaptation strategy dominates the alignment objective. Current methods cannot selectively preserve diversity where it is beneficial. Quality-adjusted diversity shows that preference-tuned models retain higher diversity among high-quality outputs, and multi-dimensional linguistic benchmarks find that larger models are often less diverse than smaller ones. Automatic diversity metrics lag behind human judgments, and sampling temperature cannot recover training-induced loss.

### Mechanisms and mitigations

DPO's gradient imbalance suppresses dispreferred responses, and likelihood displacement shifts probability to unintended outputs. KL-regularized RL specifies unimodal targets by construction, preference collapse arises from KL amplification, and chat templates induce diversity collapse. Training on recursively generated synthetic data causes progressive tail disappearance. Proposed mitigations include forward-KL optimization, entropy-constrained RL, decoupled regularization, game-theoretic SFT, diversity-aware preference optimization, and conformative decoding. A single reward function is insufficient to represent diverse human preferences.

## 3 Experimental Setup

### 3.1 Models and training lineages

We study 13 Olmo 3 checkpoints at the 7B scale. Post-training applies up to three stages, SFT, DPO, and RL, starting from the same base model.

**Base** (1 model). The base model is pretrained on Dolma 3 Mix (6T tokens), midtrained on Dolmino Mix (100B tokens), and context-extended to 65K tokens.

**Think** (3 models: Think-SFT, Think-DPO, Think). SFT trains on ~2.3M synthetic CoT reasoning traces using (prompt, completion) pairs from two teachers: QwQ-32B and DeepSeek-R1. DPO uses ~200K Delta Learning pairs. The RL stage uses a variation of GRPO with verifiable rewards and no KL penalty, and trains on ~105K prompts, to produce Think.

**Think-not-thinking**. To isolate the contribution of the CoT generation format from the learned weights, we additionally evaluate all three Think checkpoints with CoT suppressed by prefilling an empty block, forcing direct answers.

**Instruct** (3 models: Instruct-SFT, Instruct-DPO, Instruct). SFT *initializes from* Think-SFT, then trains on ~2.2M examples that include function-calling, strip reasoning traces, and draw from multiple sources (GPT-3.5, GPT-4, GPT-4.1) rather than two teachers. DPO (~260K pairs) uses the same pool of prompts as Think-DPO but with the thinking mode disabled, adding multi-turn and GPT-judged preference pairs. The same RL stage as Think produces the final Instruct model.

**RL-Zero** (6 models). Applies RL training directly to Base, bypassing SFT and DPO. Four Olmo 3 variants target different reward domains: RL-Zero-Math, RL-Zero-Code, RL-Zero-IF, and RL-Zero-General (~105K prompts each). Two additional Olmo 3.1 variants (RL-Zero-Math3.1, RL-Zero-Code3.1) are trained for more steps.

### 3.2 Tasks and Data

**Summarization.** TL;DR, CNN/DailyMail, and XSum. Bounded output length controls for length confounds, and multiple valid summaries provide a clear diversity signal.

**Code.** HumanEval, MBPP, and CRUXEval. Outputs can be syntactically different but functionally identical, and RL directly optimizes code tasks.

**Reasoning.** GSM8K, MATH-Algebra, MATH-Geometry, and TruthfulQA, the primary Think and RL-Zero training domain. Diversity here measures variation in solution *strategy* with answers held constant.

**Instruction following.** Alpaca, open-ended, and IFEval, with verifiable format constraints.

**Creative writing.** WritingPrompts, where diversity is intrinsically desirable.

**Value pluralism.** PRISM and WildBench, which test whether alignment imposes a single perspective on contested topics.

We measure training–evaluation overlap using C13-gram matching between the four Dolci post-training datasets and all fifteen evaluation tasks (Appendix J). Nine datasets show negligible overlap (≤2%). HumanEval, CRUXEval, IFEval, MATH-Algebra, MATH-Geometry, and WildBench show elevated overlap (7–30%), traceable to shared upstream data. While we flag these benchmarks, our findings on contaminated tasks are consistent with the patterns on the clean tasks.

### 3.3 Metrics

We measure diversity along four complementary axes (detailed definitions in Appendix B).

**EAD** counts unique n-grams normalized against the expected count under a uniform draw (averaged over n ∈ {1,...,5}), capturing *lexical* diversity.

**SBERT** computes mean pairwise cosine distance of sentence embeddings (all-mpnet-base-v2), capturing *semantic* diversity (0 = collapse, 1 = dissimilar). For code tasks we additionally report *semantic* diversity with UniXcoder embeddings (Appendix F).

**NLI** scores output pairs with an NLI classifier (roberta-large-mnli), following standard approaches.