Tandem Reinforcement Learning with Verifiable Rewards

# Tandem Reinforcement Learning with Verifiable Rewards
Source: [https://arxiv.org/html/2606.28166](https://arxiv.org/html/2606.28166)
Difan Jiao†∗Raghav Singhal‡Robert West‡Ashton Anderson†∗ †University of Toronto‡EPFL ∗Contact:\{difanjiao, ashton\}@cs\.toronto\.edu

###### Abstract

Reinforcement learning with verifiable rewards \(RLVR\) has significantly improved the reasoning capability of large language models, reaching expert or even superhuman performance in domains such as competition math\. However, whether weaker agents and humans can actually harness this capability is far less certain, with RLVR documented to drift reasoning toward idiosyncratic patterns such as poor readability and language mixing\. Tandem training is a recently introduced paradigm that targets this compatibility problem: a trained, stronger senior co\-generates each rollout with a frozen, weaker junior, and the two are rewarded as a team, so the senior is pushed to reason in ways the junior can follow\. Yet this paradigm has so far been demonstrated only in proof\-of\-concept settings, leaving open whether it scales to the long chains of thought of the modern RLVR pipeline\.

In this work, we propose Tandem Reinforcement Learning \(TRL\), which carries the tandem training paradigm into RLVR\. In TRL, the senior and a frozen junior alternate stochastically to co\-generate the reasoning, the resulting generation is rewarded, and the standard GRPO loss is applied to the senior\. Training Qwen3\-4B\-Instruct on competition math, we find that TRL matches vanilla GRPO on solo reasoning capability while three properties emerge together from the same rollout structure: stronger handoff robustness with the junior, reduced distributional drift from the junior, and a chain\-of\-thought more legible to the junior\. Our results demonstrate a promising route for RLVR with practical payoffs in multi\-model communication and human compatibility\.111The codebase is available at[https://github\.com/CSSLab/Tandem\-RLVR](https://github.com/CSSLab/Tandem-RLVR)\.

## 1Introduction

Reinforcement learning with verifiable rewards \(RLVR\) has emerged as a dominant post\-training paradigm in language\-model post\-training\. In RLVR, a model samples a solution, an external verifier scores the final answer, and policy optimization reinforces successful trajectories\. This simple loop has proved remarkably effective, producing large gains on mathematical and competition\-style benchmarks and eliciting long, self\-correcting chains of thought without process supervision\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib3); Guoet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib22); Yuet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib27)\)\.

However, this objective has an important drawback: it can improve benchmark performance without requiring the reasoning trajectory to remain compatible with weaker agents\. A model can become better at solving problems while moving toward reasoning patterns that its pre\-RL base model, weaker partner models, or human overseers are less able to predict, continue, or understand\. This concern is not hypothetical: RLVR is known to induce substantial distributional drift from the base policy\(Guoet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib22); Liet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib37); Menget al\.,[2026](https://arxiv.org/html/2606.28166#bib.bib36)\), and recent work suggests that some reasoning behavior may become concentrated in idiosyncratic token patterns rather than transparent surface explanations\(Kirchneret al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib48); Karpovet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib31); Skafet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib32)\)\. For oversight and multi\-agent systems, this is a serious weakness, since a model’s reasoning is often useful only insofar as weaker humans, monitors, or partner models can still follow and act on it\(Burnset al\.,[2023](https://arxiv.org/html/2606.28166#bib.bib34); Lightmanet al\.,[2023](https://arxiv.org/html/2606.28166#bib.bib5); Davidsonet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib28)\)\.

Standard defenses against this drift, such as KL penalties against a reference policy\(Ziegleret al\.,[2019](https://arxiv.org/html/2606.28166#bib.bib43); Ouyanget al\.,[2022](https://arxiv.org/html/2606.28166#bib.bib25)\), supervised distillation\(Hintonet al\.,[2015](https://arxiv.org/html/2606.28166#bib.bib45); Magisteret al\.,[2023](https://arxiv.org/html/2606.28166#bib.bib46)\), or process supervision\(Uesatoet al\.,[2022](https://arxiv.org/html/2606.28166#bib.bib47); Lightmanet al\.,[2023](https://arxiv.org/html/2606.28166#bib.bib5)\), require that the designer to commit, in advance, to an explicit specification of what “intelligible” reasoning looks like, encoded either as a reference distribution or as labeled traces\. Outside narrow deployment contexts, such a specification is difficult to write down\(Baiet al\.,[2022](https://arxiv.org/html/2606.28166#bib.bib4); Lightmanet al\.,[2023](https://arxiv.org/html/2606.28166#bib.bib5)\)\. Tandem training, a recently\-introduced training paradigm\(Hamadeet al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib2); Westet al\.,[2026](https://arxiv.org/html/2606.28166#bib.bib1)\), offers a potential solution\. Rather than penalize distance from a fixed reference, a trained*senior*model co\-generates every rollout with a frozen*junior*partner model, and the team is rewarded as a whole\. A successful trajectory is then, by construction, one the junior could have continued\. Intelligibility is thus operationalized through outcomes rather than declared up front\. Prior work has demonstrated proofs of concept in chess\(Hamadeet al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib2)\)and in a simplified reasoning setting\(Westet al\.,[2026](https://arxiv.org/html/2606.28166#bib.bib1)\), but it remains unclear whether the paradigm can provide the intelligibility within the modern RLVR pipeline\.

![Refer to caption](https://arxiv.org/html/2606.28166v1/x1.png)Figure 1:Tandem Reinforcement Learning \(TRL\) at a glance\.In this work, we introduce Tandem Reinforcement Learning, which carries the tandem training paradigm into GRPO\-style RLVR\. As illustrated in Figure[1](https://arxiv.org/html/2606.28166#S1.F1), a trainable senior and a frozen junior, both initialized from the same base model, co\-generate each response by stochastically alternating at word boundaries\. The completed response receives the usual binary verifier reward, and the senior is updated with the standard GRPO objective on the tokens it emitted\. Thus, TRL only changes how rollouts are produced, not the reward, verifier, or policy\-gradient loss, and differences between TRL\- and GRPO\-trained seniors are attributable to rollout structure alone\.

Training Qwen3\-4B\-Instruct on competition math, we find that three properties emerge together from this single change to rollout structure\. First, compared with a matched GRPO baseline, TRL preserves solo reasoning capability across competition benchmarks\. Second, we find that TRL improves handoff robustness\. When paired at inference with a frozen junior under a reasoning\-step schedule, the TRL senior outperforms the GRPO senior by up to \+6\.6 pass@8 points on AIME\. Third, distributional drift is substantially curbed and legibility improves\. TRL’s marginal token distribution stays closer to the base model’s distribution across the vocabulary \(14% lower KL divergence\); among the 500 tokens GRPO most displaces, 87% shift back toward the base under TRL; and the senior’s chain\-of\-thought becomes more legible to the base model, with per\-token cross\-entropy under the junior dropping up to 17%\. An ablation that adds a per\-token KL penalty toward the junior on top of vanilla GRPO does not reproduce these gains, showing they come from the tandem rollout structure rather than from regularization toward the junior\.

Our results show that compatibility with a weaker partner, distributional anchoring to a base model, and full RLVR capability are obtainable from the same intervention on rollout structure, without modifying the reward, the verifier, or the loss\. This points to rollout structure as an underexplored design axis for the RLVR pipeline, with practical payoffs for multi\-model communication and human compatibility\.

## 2Related work

Tandem training\.Tandem training was first introduced byHamadeet al\.\([2024](https://arxiv.org/html/2606.28166#bib.bib2)\)in chess, where they showed that optimizing for partner compatibility, i\.e\., winning a team game alongside a weaker collaborator, is a distinct objective from optimizing for raw ability\. InWestet al\.\([2026](https://arxiv.org/html/2606.28166#bib.bib1)\), this tandem training paradigm is carried into language modeling, demonstrating on GSM8K that randomized handoffs to a frozen junior during RL teach a stronger senior to abandon jargon and adapt its language to weaker partners while keeping task accuracy high\.

Reinforcement learning with verifiable rewards \(RLVR\)\.Recent advances in reasoning post\-training have leveraged reinforcement learning with verifiable rewards \(RLVR\)\.Shaoet al\.\([2024](https://arxiv.org/html/2606.28166#bib.bib3)\)introduced Group Relative Policy Optimization \(GRPO\) as a practical recipe for this setting, andGuoet al\.\([2025](https://arxiv.org/html/2606.28166#bib.bib22)\)demonstrated that outcome\-only RL can elicit strong reasoning capabilities without process supervision\. A growing follow\-up literature expands the GRPO design space along two axes: optimization\-side variants revisit stability, optimization biases, and ratio aggregation\(Yuet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib27); Liuet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib21); Zhenget al\.,[2025a](https://arxiv.org/html/2606.28166#bib.bib16); Zhaoet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib17); Chuet al\.,[2025a](https://arxiv.org/html/2606.28166#bib.bib18); Zenget al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib19)\), while rollout\-side work targets sampling efficiency, prompt filtering, and replay of high\-signal trajectories\(Zhanget al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib38); Zhanet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib39); Zhenget al\.,[2025b](https://arxiv.org/html/2606.28166#bib.bib40)\)\. TRL operates at a finer granularity than either axis: it modifies the*rollout structure*at the level of who emits each token within a single rollout, leaving both the loss aggregation and the rollout\-management strategy unchanged\.

RLVR\-induced distributional drift\.A documented empirical regularity of RLVR is that the trained policy drifts measurably away from the pretrained base distribution\. DeepSeek\-R1\-Zero, a pure outcome\-only RLVR model, exhibits poor readability and unprompted language mixing as a side effect of training\(Guoet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib22)\)\.Liet al\.\([2025](https://arxiv.org/html/2606.28166#bib.bib37)\)identify RLVR specifically as the post\-training stage that triggers Chinese–English code\-switching in bilingual reasoning models, and recent work has further uncovered steganographic chain\-of\-thought patterns in which reasoning becomes load\-bearing yet undetectable to a downstream monitor\(Kirchneret al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib48); Karpovet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib31); Skafet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib32)\)\. Such drift also carries downstream costs, as recent work finds that compatible token\-level distributions between teacher and student are a governing condition for successful on\-policy distillation\(Liet al\.,[2026](https://arxiv.org/html/2606.28166#bib.bib49)\)\. On the other hand, a recent complementary line of work argues that not all drift is unhelpful\.Menget al\.\([2026](https://arxiv.org/html/2606.28166#bib.bib36)\)show through token\-level cross\-sampling interventions that a small subset of high\-divergence positions is functionally responsible for RLVR’s reasoning gains\.

Tandem training has so far been demonstrated only in proof\-of\-concept settings outside the RLVR pipeline\. We carry the paradigm into RLVR, the mainstream post\-training framework behind current reasoning LLMs, and observe how the resulting model behaves both on its own and in cooperation with weaker collaborators\.

## 3Tandem reinforcement learning

### 3\.1Preliminaries: tandem training

Tandem training\(Hamadeet al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib2); Westet al\.,[2026](https://arxiv.org/html/2606.28166#bib.bib1)\)is a recently\-introduced training paradigm in which two language models jointly produce each rollout\. A trainable*senior*policyπsen\\pi\_\{\\text\{sen\}\}and a frozen*junior*policyπjun\\pi\_\{\\text\{jun\}\}stochastically alternate generations, with coin flips determining the next active model at predetermined handoff boundaries such as tokens, words, or sentences \(e\.g\. if we choose to alternate at the token level, for every token we flip a coin to determine which model generates the following token\)\. The co\-constructed rollout that emerges is scored by a single reward function as one trajectory, and the senior is updated against that reward using a policy\-gradient algorithm while the junior remains unchanged\.

Tandem training is designed to produce more*compatible*models\. An operational definition of compatibility, or intelligibility, is*handoff robustness*\(Westet al\.,[2026](https://arxiv.org/html/2606.28166#bib.bib1)\): a model’s output is intelligible to another agent if that agent can continue it without derailing the trajectory\. If a tandem training rollout ends successfully, the senior must have been generating in such a way that the junior could continue without crashing the trajectory\. Reinforcing such rollouts therefore selects for senior behavior that the junior can successfully collaborate with, thus promoting compatibility\. In this setup, no explicit definition of “intelligibility” is required—since we are working with verifiable rewards, we can directly measure how successful the senior\-junior tandem team is at any given point\. This is appealing because intelligibility is otherwise hard to codify: explicit methods such as system prompts or supervised finetuning on canonical solutions each demand an*a priori*specification that depends on deployment context\(Westet al\.,[2026](https://arxiv.org/html/2606.28166#bib.bib1); Baiet al\.,[2022](https://arxiv.org/html/2606.28166#bib.bib4); Lightmanet al\.,[2023](https://arxiv.org/html/2606.28166#bib.bib5)\)\.

In this work, we generalize the tandem training paradigm to Reinforcement Learning with Verifiable Rewards \(RLVR\), the cornerstone of contemporary large reasoning models\. We refer to our instantiation as Tandem Reinforcement Learning \(TRL\)\. To adapt to RLVR, we make three design choices: the senior and junior are initialized from the same base model \(§[3\.2](https://arxiv.org/html/2606.28166#S3.SS2)\); handoffs happen at the word level \(§[3\.3](https://arxiv.org/html/2606.28166#S3.SS3)\); and the senior is updated in the same way as Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib3)\)on the senior\-emitted tokens \(§[3\.4](https://arxiv.org/html/2606.28166#S3.SS4)\)\. We now discuss each of these choices in turn\.

### 3\.2Tandem pairs

In general tandem training, the senior agent is typically stronger, or at least stylistically distinct, from the junior, so that the senior must adapt to be compatible with a partner with weaker capabilities\. In this work adapting tandem training to RLVR, such asymmetric pairings remain available, but we begin with the natural choice of setting the junior as the senior’s own pre\-RL initialization—before training, the junior and senior models begin as identical copies of each other\. This self\-pairing offers methodological benefits\. In a stronger\-weaker pair, the trained senior is encouraged to learn how to adapt to potentially a very different style\. With identical capabilities at the start, this pressure is eliminated, and instead the senior can focus on improving its capabilities while minimizing drift from its initial state, thus more directly addressing our motivation of developing a capable, compatible, and intelligible model\. We therefore initializeπsen\\pi\_\{\\text\{sen\}\}andπjun\\pi\_\{\\text\{jun\}\}from the same base model and freezeπjun\\pi\_\{\\text\{jun\}\}for the duration of training\.

### 3\.3Tandem rollout

A tandem rollout produces a shared responsey1:Ty\_\{1:T\}from promptxx, where each stepttis generated by an active modelat∈\{sen,jun\}a\_\{t\}\\in\\\{\\text\{sen\},\\text\{jun\}\\\}\. At each step, both models receive the contextx⋅y<tx\\cdot y\_\{<t\}, the active model samplesyt∼πat\(⋅∣x⋅y<t\)y\_\{t\}\\sim\\pi\_\{a\_\{t\}\}\(\\cdot\\mid x\\cdot y\_\{<t\}\), and the chosenyty\_\{t\}is appended and then fed to both models so that both states condition on the same history\. FollowingWestet al\.\([2026](https://arxiv.org/html/2606.28166#bib.bib1)\), the active model is redrawn by an independent fair coin flip at every word boundary: ifyty\_\{t\}begins a new orthographic word,at\+1a\_\{t\+1\}is set to the senior with probabilityppand to the junior otherwise; ifyty\_\{t\}does not begin a new word,at\+1=ata\_\{t\+1\}=a\_\{t\}\. We usep=0\.5p=0\.5in consistency with original tandem training\. Details of this setup are included in Appendix[A\.1](https://arxiv.org/html/2606.28166#A1.SS1)\.

We choose to stochastically alternate at word\-level granularity over alternative choices \(e\.g\., token\-level or sentence\-level\)\. First, at sentence\- or paragraph\-level the senior can dominate a rollout simply by suppressing boundary tokens \(e\.g\., end\-of\-sentence or end\-of\-line tokens\) that would force a potential handoff222This reward\-hacking failure mode is induced by the schedule itself: as the senior strengthens, suppressing boundary tokens becomes a low\-cost route to extending its own control and avoiding handoffs\., collapsing the schedule back to vanilla GRPO\. Second, at token level, alternation at subword granularity disrupts coherent utterances before they form\. In contrast, word\-level boundaries fire on every natural\-language word and are difficult to suppress without sacrificing fluency, making them the coarsest granularity the senior cannot trivially erase by output choice\.

Algorithm 1Tandem Reinforcement Learning \(TRL\)\.0:Base model

π0\\pi\_\{0\}; training set

𝒟\\mathcal\{D\}; batch size

BB; group size

GG; max response length

LL; word\-boundary set

ℬ\\mathcal\{B\}; subword\-span cap

KK; Bernoulli probability

pp; verifier

r​\(⋅\)r\(\\cdot\)
1:Initialize

πsen←π0\\pi\_\{\\text\{sen\}\}\\leftarrow\\pi\_\{0\}and

πjun←π0\\pi\_\{\\text\{jun\}\}\\leftarrow\\pi\_\{0\}; freeze

πjun\\pi\_\{\\text\{jun\}\}
2:for

τ=1,2,…\\tau=1,2,\\ldotsdo

3:Sample a batch of prompts

\{xj\}j=1B∼𝒟\\\{x\_\{j\}\\\}\_\{j=1\}^\{B\}\\sim\\mathcal\{D\}
4:foreach

\(j,i\)\(j,i\)with

j∈\[ℬ\]j\\in\[\\mathcal\{B\}\]and

i∈\[G\]i\\in\[G\]do

5:// Tandem rollout \(§[3\.3](https://arxiv.org/html/2606.28166#S3.SS3)\)

6:Draw

a1∼Bernoulli​\(p\)a\_\{1\}\\sim\\mathrm\{Bernoulli\}\(p\)over

\{sen,jun\}\\\{\\text\{sen\},\\text\{jun\}\\\};

c←0c\\leftarrow 0
7:for

t=1,…,Lt=1,\\ldots,Ldo

8:Forward

πsen\\pi\_\{\\text\{sen\}\}and

πjun\\pi\_\{\\text\{jun\}\}on

xj⋅y<tx\_\{j\}\\cdot y\_\{<t\}; sample

ytsen,ytjuny\_\{t\}^\{\\text\{sen\}\},y\_\{t\}^\{\\text\{jun\}\}independently

9:

yt←ytaty\_\{t\}\\leftarrow y\_\{t\}^\{a\_\{t\}\};

mt←𝟙​\[at=sen\]m\_\{t\}\\leftarrow\\mathbbm\{1\}\[a\_\{t\}=\\text\{sen\}\]
10:breakif

yty\_\{t\}is the end\-of\-sequence token

11:if

yt∈ℬy\_\{t\}\\in\\mathcal\{B\}or

c≥Kc\\geq Kthen

12:Draw

at\+1∼Bernoulli​\(p\)a\_\{t\+1\}\\sim\\mathrm\{Bernoulli\}\(p\)over

\{sen,jun\}\\\{\\text\{sen\},\\text\{jun\}\\\};

c←0c\\leftarrow 0
13:else

14:

at\+1←ata\_\{t\+1\}\\leftarrow a\_\{t\};

c←c\+1c\\leftarrow c\+1
15:endif

16:endfor

17:Record

y\(j,i\)←y1:ty^\{\(j,i\)\}\\leftarrow y\_\{1:t\}and authorship mask

m\(j,i\)←m1:tm^\{\(j,i\)\}\\leftarrow m\_\{1:t\}
18:endfor

19:// Senior\-only GRPO update \(§[3\.4](https://arxiv.org/html/2606.28166#S3.SS4)\)

20:Compute rewards

r\(j,i\)←r​\(y\(j,i\)\)r^\{\(j,i\)\}\\leftarrow r\(y^\{\(j,i\)\}\)
21:Compute group\-relative advantages

At\(j,i\)A^\{\(j,i\)\}\_\{t\}from

\{r\(j,i\)\}i=1G\\\{r^\{\(j,i\)\}\\\}\_\{i=1\}^\{G\}
22:Update

πsen\\pi\_\{\\text\{sen\}\}with the GRPO objective of §[3\.4](https://arxiv.org/html/2606.28166#S3.SS4), with response mask narrowed by

m\(j,i\)m^\{\(j,i\)\}
23:endfor

### 3\.4Tandem policy update

Each tandem rollout receives a single binary rewardr​\(y\)∈\{0,1\}r\(y\)\\in\\\{0,1\\\}from a math verifier, computed on the full responsey1:Ty\_\{1:T\}regardless of which model emitted each token\. The senior is updated with the standard GRPO objective\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib3)\), with the response mask set to select only senior\-emitted positions; the per\-token policy\-gradient contribution is

ℒTRL​\(s​e​n\)=−𝔼​\[∑t:at=senAt​log⁡πsen​\(yt∣x⋅y<t\)\],\\mathcal\{L\}\_\{\\text\{TRL\}\}\(sen\)\\;=\\;\-\\,\\mathbb\{E\}\\\!\\left\[\\;\\sum\_\{t\\,:\\,a\_\{t\}=\\text\{sen\}\}A\_\{t\}\\,\\log\\pi\_\{\\text\{sen\}\}\(y\_\{t\}\\mid x\\cdot y\_\{<t\}\)\\;\\right\],\(1\)
whereAtA\_\{t\}is the standard GRPO group\-relative advantage; the clipped surrogate ratio and KL term are also identical to GRPO and omitted here for brevity\. Junior\-emitted tokens enter the response, and so shape the reward the senior receives, but contribute no gradient\.

Specifically, while the original tandem\-training formulation also includes a soft junior\-imitation term that pullsπsen\\pi\_\{\\text\{sen\}\}towardπjun\\pi\_\{\\text\{jun\}\}on junior\-emitted positions with coefficientλjun\\lambda\_\{\\text\{jun\}\}, we setλjun=0\\lambda\_\{\\text\{jun\}\}=0\. Beyond removing a hyperparameter we have no principled value for, this keeps the per\-token loss formally the same as vanilla GRPO\. Thus, any difference between TRL\- and GRPO\-trained seniors is then attributable to the rollout structure that producesyy, and not to an auxiliary regularization term\.

The end\-to\-end TRL pipeline is illustrated in Algorithm[1](https://arxiv.org/html/2606.28166#alg1)\.

## 4Results

### 4\.1Experimental setup

We fine\-tune Qwen3\-4B\-Instruct\-2507\(Yanget al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib10)\)on DeepScaleR\(Tanet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib7)\)with a binary correctness reward on the boxed final answer\. Our baseline is vanilla GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib3)\)on the same base model, with optimization settings shared with TRL \(Appendix[A\.3](https://arxiv.org/html/2606.28166#A1.SS3)\)\. We evaluate on AMC 23–25, AIME 24–26, and Minerva Math\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2606.28166#bib.bib11)\), and report pass@kkvia an unbiased estimator followingChenet al\.\([2021](https://arxiv.org/html/2606.28166#bib.bib8)\)\. We estimate bootstrap standard errors by resampling evaluation problems, reported as±\\pmvalues in tables and as shaded bands in figures\.

This section is organized in four parts\. In §[4\.2](https://arxiv.org/html/2606.28166#S4.SS2), we measure the resulting models’ solo reasoning capabilities, examining how TRL influences the reasoning gains RLVR delivers, with vanilla GRPO as the reference point\. We then turn to handoff robustness in §[4\.3](https://arxiv.org/html/2606.28166#S4.SS3), the central behavioural property tandem training was originally proposed to elicit; we pair the trained models with the junior model at inference time and evaluate the team’s performance\. In §[4\.4](https://arxiv.org/html/2606.28166#S4.SS4)we evaluate RL\-induced distribution shift, a documented cost of RLVR, examining whether TRL’s co\-generation structure measurably resists it\. Finally, in §[4\.5](https://arxiv.org/html/2606.28166#S4.SS5)we measure whether this distributional anchoring makes the senior’s outputs more legible to the junior\.

![Refer to caption](https://arxiv.org/html/2606.28166v1/x2.png)Figure 2:Reasoning capabilities \(measured by pass@kk,↑\\uparrow\) on mathematical reasoning benchmarks for Qwen3\-4B\-Instruct and its GRPO\- and TRL\-trained seniors\.![Refer to caption](https://arxiv.org/html/2606.28166v1/x3.png)Figure 3:Reasoning\-step handoff robustness \(measured by pass@​k@k,↑\\uparrow\) on mathematical reasoning benchmarks of GRPO and TRL seniors paired with the junior, frozen Qwen3\-4B\-Instruct\.
### 4\.2TRL retains the reasoning gains of RLVR

We first measure each model’s solo reasoning capability, with no junior in the loop at test time, to examine how the tandem rollout structure influences the gains RLVR delivers\. Each senior generatesnnindependent samples per problem; we report pass@kkvia the unbiased estimator fork∈\{1,2,4,8,16,32\}k\\in\\\{1,2,4,8,16,32\\\}\.

Figure[2](https://arxiv.org/html/2606.28166#S4.F2)shows pass@kkcurves for the pre\-RL base, vanilla GRPO, and TRL across benchmarks and their macro average\. Two observations stand out\. First, both trained seniors lift the base across the fullkkrange on all three benchmarks, confirming that RLVR is delivering a non\-trivial capability gain on this base\. Second, the GRPO and TRL curves track each other closely throughout, with differences at any givenkkwithin typical run\-to\-run variance and no consistent direction across benchmarks\. We read this as empirical evidence that the tandem rollout structure does not cost the senior any*solo*capability: the gains RLVR delivers under vanilla GRPO are retained under TRL\.

### 4\.3TRL offers better handoff robustness

In §[3\.3](https://arxiv.org/html/2606.28166#S3.SS3)we argued against training with sentence\- or paragraph\-level handoffs, on the grounds that the senior could game such schedules by suppressing the boundary tokens that would force a switch\. For evaluation, however, there is no risk of this type of gaming behavior, and we want to measure how cooperation between language models actually unfolds in practice\. In multi\-agent reasoning systems, such as drafter\-target setups for inference and human\-AI handoffs, the natural unit of communication is a reasoning step rather than an individual word, with one party producing a step and a partner picking up where it left off\(Chuet al\.,[2025b](https://arxiv.org/html/2606.28166#bib.bib29); Wanget al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib30); Davidsonet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib28)\)333also resembling a*ply*of chess games inHamadeet al\.\([2024](https://arxiv.org/html/2606.28166#bib.bib2)\)\.\. This motivates our evaluation of handoff robustness under a reasoning\-step schedule: at inference, the senior and the junior alternate at every\\n\\n\-boundary token, each producing one reasoning step before yielding control to the other, with the team\-produced response scored as a whole\.

Figure[3](https://arxiv.org/html/2606.28166#S4.F3)shows pass@kkfork∈\{1,2,4,8\}k\\in\\\{1,2,4,8\\\}\. TRL leads on all three benchmarks, with the advantage most pronounced on AIME:\+4\.6\+4\.6points atk=1k\{=\}1\(29\.329\.3vs\.24\.724\.7\) widening to\+6\.6\+6\.6atk=8k\{=\}8\(44\.444\.4vs\.37\.837\.8\)\. The macro\-average gap grows from\+2\.2\+2\.2atk=1k\{=\}1to\+3\.1\+3\.1atk=8k\{=\}8\. Read together with §[4\.2](https://arxiv.org/html/2606.28166#S4.SS2), the picture is clean: the TRL senior is no less capable than the GRPO senior reasoning alone, and is the stronger half of a team\. We confirm the statistical significance of this gap, and also of the further TRL gains reported in following subsections, with pairedtt\-tests in §[5\.3](https://arxiv.org/html/2606.28166#S5.SS3)\.

### 4\.4TRL resists RLVR\-induced distribution shift

TRL’s rollout structure carries an implicit consequence beyond handoff robustness\. Because the junior is a frozen copy of the senior’s pre\-RL base, every successful tandem rollout is one the junior could have continued at any word boundary\. This creates a structural pressure against token choices that lie far outside the junior’s predictive support\. Whether this pressure measurably anchors the senior’s output distribution is an empirical question and one worth asking, as distributional drift is a documented cost of RLVR at scale\. Models develop idiosyncratic token patterns that diverge from the pre\-RL base, including language mixing and syntactic irregularities\(Guoet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib22); Yuet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib27)\), and recent work has uncovered steganographic chain\-of\-thought patterns whose reasoning is no longer recoverable from surface text\(Karpovet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib31); Skafet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib32)\)\. Such drift undermines oversight, as supervision requires the stronger model’s reasoning to remain within the interpretive reach of weaker overseers\(Burnset al\.,[2023](https://arxiv.org/html/2606.28166#bib.bib34); Lightmanet al\.,[2023](https://arxiv.org/html/2606.28166#bib.bib5)\)\.

![Refer to caption](https://arxiv.org/html/2606.28166v1/x4.png)Figure 4:Distributional deviation from the base model for GRPO and TRL\.*Left*: survival curves of absolute per\-token log\-ratio to the base over tokens\.*Right*: for the top\-500 most displaced tokens by GRPO, we show the drift versus TRL recovery\.We therefore ask whether TRL’s co\-generation structure measurably limits this drift\. To operationalize, we estimate each model’s marginal token distribution from all senior generations on our evaluation benchmarks\. Per\-token deviation from the junior is then measured by the signed log\-ratiolog⁡\(psen​\(t\)/pjun​\(t\)\)\\log\(p\_\{\\mathrm\{sen\}\}\(t\)\\,/\\,p\_\{\\mathrm\{jun\}\}\(t\)\)for each tokent∈𝒱t\\in\\mathcal\{V\}\(vocabulary space\), positive when the senior over\-usesttrelative to the junior and negative when it under\-uses it\.

Figure[4](https://arxiv.org/html/2606.28166#S4.F4)shows that TRL curbs this drift\. The left panel plots the survival curve of absolute deviation over all tokens appearing frequently in the junior’s own outputs\. As shown, TRL’s curve lies uniformly below Vanilla GRPO’s at every threshold, indicating that fewer TRL tokens stray far from the junior at any scale\. Full\-vocabulary KL\-divergence\(Kullback,[1997](https://arxiv.org/html/2606.28166#bib.bib33)\)confirms the aggregate, where TRL is 14% closer to the junior by KL \(0\.0220\.022vs\.0\.0260\.026\)\. The right panel zooms into the 500 tokens with the largest GRPO drift444tokens with highestlog⁡\(pGRPO​\(t\)/pjun​\(t\)\)\\log\(p\_\{\\mathrm\{GRPO\}\}\(t\)\\,/\\,p\_\{\\mathrm\{jun\}\}\(t\)\)\.\. Among these, 87% show TRL shifting in the opposing direction: under\-emitting what GRPO over\-emits and vice versa \(Spearmanρ=−0\.58\\rho=\{\-\}0\.58,p<10−275p<10^\{\-275\}\), pushing back toward the junior’s distribution\. Taken together, TRL stays closer to the junior than Vanilla GRPO: globally in aggregate, and on the tokens GRPO displaces most\. A more detailed qualitative analysis of drifted tokens is included in Appendix[B\.1](https://arxiv.org/html/2606.28166#A2.SS1)\.

### 4\.5TRL yields a more junior\-legible chain\-of\-thought

TRL’s distributional anchoring raises a question: does the junior \(base\) model find the senior’s reasoning more predictable? §[4\.4](https://arxiv.org/html/2606.28166#S4.SS4)establishes that the senior’s overall token usage stays nearer the junior’s, but it does not directly speak to whether the junior, reading the senior’s chain\-of\-thought left\-to\-right, would find the next token predictable at each step\. The two are correlated but separable: a marginal\-similar policy can still be conditionally surprising at most positions\.

In this part, we measure this junior legibility with two analyses, summarized in Table[1](https://arxiv.org/html/2606.28166#S4.T1)\. The first is the junior’s per\-token cross\-entropy on the senior’s chain\-of\-thought: the average*nats*required to encode each senior token under the junior’s predictive distribution, where lower means the junior is less surprised\. TRL’s cross\-entropy is lower by0\.0100\.010nats on average, with clear gains on AMC and Minerva and a near match on AIME\. Specifically, a reduction of0\.0100\.010nats represents a7\.6%7\.6\\%decrease in per\-token surprisal on average, reaching17%17\\%on Minerva, meaning the junior finds TRL’s chain\-of\-thought that much easier to follow at each step\.

The second is the token\-level distribution overlapα=∑vmin⁡\(psen​\(v∣⋅\),pjun​\(v∣⋅\)\)\\alpha=\\sum\_\{v\}\\min\(p\_\{\\mathrm\{sen\}\}\(v\\mid\\mathrm\{\\cdot\}\),\\,p\_\{\\mathrm\{jun\}\}\(v\\mid\\mathrm\{\\cdot\}\)\)averaged over all positions\(Leviathanet al\.,[2023](https://arxiv.org/html/2606.28166#bib.bib35)\): the probability mass both models assign to the same tokens at each step, ranging from0\(fully disjoint\) to11\(identical\)\. As shown, TRL’s distribution overlap is uniformly higher across all three benchmarks\. Note the absolute gap between0\.9730\.973and0\.9600\.960may appear small, but expressed as shortfall from perfect agreement, TRL’s gap is around0\.030\.03against Vanilla GRPO’s0\.040\.04: a roughly30%30\\%reduction in the fraction of positions where the two models disagree\.

Table 1:Junior legibility of senior generated chain\-of\-thoughts \(Qwen3\-4B\-Instruct junior\)\.Boldmarks the better model per cell\. Subscripts report±1\\pm 1standard error of the mean over 10,000 bootstrap resamples of problems \(stratified by benchmark for the macro average\)\.

## 5Discussion

### 5\.1Training dynamics of TRL

TRL replaces every rollout with a joint two\-model generation, so it is natural to ask whether this structural change disrupts training or imposes prohibitive overhead\. Figure[5](https://arxiv.org/html/2606.28166#S5.F5)reports the core training signals\. TRL is operationalized end\-to\-end: checkpoint selection is driven by tandem\-rollout accuracy rather than solo accuracy, so the two curves in panel \(d\) are plotted on separate axes to reflect that they measure different evaluation protocols\. This is intentional, as selecting by solo accuracy would be inconsistent with the cooperative objective TRL optimizes\.

Panel \(a\) shows that training reward follows a stable trajectory under TRL, closely tracking GRPO throughout\. Panel \(b\) shows that response lengths evolve similarly in both runs, without runaway growth or collapse to short responses, suggesting the senior neither offloads the full generation burden to the junior nor exploits the joint rollout to terminate early\. Together, these signals indicate that the tandem rollout does not destabilize the basic RL learning dynamic\.

On training cost, running two full model forwards per step is an unavoidable overhead\. A naive implementation cannot sustain the long contexts that RLVR requires and is impractical beyond toy settings \(Appendix[A\.1](https://arxiv.org/html/2606.28166#A1.SS1)\); we therefore surgically extend the vLLM backend\(Kwon,[2025](https://arxiv.org/html/2606.28166#bib.bib6)\)to both enable and accelerate TRL, reducing per\-step latency to approximately2×2\\timesthat of a single\-model run, which we regard as a standalone technical contribution of this work\. Empirically, TRL reaches its best checkpoint at step120 compared to step200 for GRPO, partially offsetting the per\-step overhead; accumulated wall\-clock to convergence is 9\.4 hours for TRL versus 7\.8 hours for GRPO \(panel c\)\.

We also observe a sharp performance decline in TRL after sufficient training steps, a dynamic thatWestet al\.\([2026](https://arxiv.org/html/2606.28166#bib.bib1)\)similarly report for tandem training in language generation\. We note that this pattern is consistent with what one might expect from prolonged co\-generation training: as the senior diverges further from the frozen junior, the pressure that anchored its earlier checkpoints becomes harder to satisfy, and tandem rollouts may begin to fail more frequently\. However, this does not affect the quality of the TRL senior at its best checkpoint, which is the model evaluated throughout §[4](https://arxiv.org/html/2606.28166#S4), and we leave theoretical and mechanistic understanding of this behavior to future work\.

![Refer to caption](https://arxiv.org/html/2606.28166v1/x5.png)Figure 5:Training dynamics of TRL and GRPO\. \(a\) Mean reward\. \(b\) Average response length for rollouts\. \(c\) Per\-step wallclock \(left axis\) and accumulated wallclock to best checkpoints \(right axis, shaded\)\. \(d\) Macro\-average benchmark accuracy under each run’s evaluation protocol\.
### 5\.2Ablation study: KL\-regularization towards the junior

TRL can be thought of as implicitly regularizing the senior toward the junior: every successful tandem trajectory is one the junior could have continued, so the training signal pushes the senior to stay near the junior’s distribution\. KL regularization toward the junior is the standard explicit way to apply this kind of anchoring in RLHF, which makes it the natural ablation baseline\. We call this baseline*KL\-Reg*, with the reference fixed at the pre\-RL base \(the same checkpoint that serves as TRL’s frozen junior\)\. Training data, optimizer settings, and evaluation protocol match TRL and vanilla GRPO\.

Table[2](https://arxiv.org/html/2606.28166#S5.T2)reports macro averages across the four evaluation axes of §[4](https://arxiv.org/html/2606.28166#S4)\. On solo capability, all three models are at parity\. On handoff robustness, however, KL\-Reg sits essentially at the vanilla GRPO level, while TRL leads by roughly three percentage points: adding a KL penalty toward the junior does not, on its own, make the senior produce text the junior can pick up and continue\. On distribution overlap and junior legibility, KL\-Reg moves partway from GRPO toward TRL but does not close the gap on either; TRL produces a chain\-of\-thought that the junior finds both more overlapping with its own distribution and more predictable token\-by\-token, by a margin KL\-Reg does not match\. We note that the KL regularization toward the junior and handoff success are correlated but structurally distinct: a senior can sit at low average KL to the junior but still produce tokens the junior cannot continue at handoff boundaries\. Raising the regularization coefficient alone cannot resolve this tension, since past a moderate value it also erodes the senior’s reasoning capability gained from RLVR\. We provide detailed discussion in Appendix[C](https://arxiv.org/html/2606.28166#A3)\.

Table 2:Our ablation study: vanilla GRPO, junior\-policy\-regularized GRPO \(KL\-Reg\), and TRL on the four quantitative evaluation axes of §[4](https://arxiv.org/html/2606.28166#S4)\.
### 5\.3Statistical significance of TRL gains

We complement the bootstrap standard errors of §[4](https://arxiv.org/html/2606.28166#S4)with paired one\-sidedtt\-tests on per\-problem statistics, mean\-pooled across the benchmarks\. For pass@kkwe test per\-problem unbiased estimates\(Chenet al\.,[2021](https://arxiv.org/html/2606.28166#bib.bib8)\); for per\-token cross\-entropy and distribution overlapα\\alphawe average per\-problem measurements before testing\. We note that each test is one\-sided in the claimed direction, and asterisks denote significance levels:p∗<0\.05\{\}^\{\*\}p<0\.05,p∗∗<0\.01\{\}^\{\*\*\}p<0\.01,p∗⁣∗∗<0\.001\{\}^\{\*\*\*\}p<0\.001\.

Table 3:Pairedtt\-tests for the evaluation gains of TRL over baselines\.ComparisonΔ\\DeltattppHandoff Robustness\(pass@kk,Δ\\Deltain percentage points,↑\\uparrow\)TRL\>\>GRPO,k=1k\{=\}1\+1\.42\+1\.42\+2\.61\+2\.610\.005∗∗0\.005^\{\*\*\}TRL\>\>GRPO,k=2k\{=\}2\+1\.72\+1\.72\+2\.66\+2\.660\.004∗∗0\.004^\{\*\*\}TRL\>\>GRPO,k=4k\{=\}4\+1\.88\+1\.88\+2\.32\+2\.320\.010∗0\.010^\{\*\}TRL\>\>GRPO,k=8k\{=\}8\+2\.07\+2\.07\+1\.83\+1\.830\.034∗0\.034^\{\*\}TRL\>\>KL\-Reg,k=1k\{=\}1\+1\.01\+1\.01\+1\.73\+1\.730\.043∗0\.043^\{\*\}TRL\>\>KL\-Reg,k=2k\{=\}2\+1\.24\+1\.24\+1\.90\+1\.900\.029∗0\.029^\{\*\}TRL\>\>KL\-Reg,k=4k\{=\}4\+1\.62\+1\.62\+2\.13\+2\.130\.017∗0\.017^\{\*\}TRL\>\>KL\-Reg,k=8k\{=\}8\+2\.07\+2\.07\+2\.14\+2\.140\.016∗0\.016^\{\*\}Junior Legibility\(per\-token cross\-entropy,Δ\\Deltain nats,↓\\downarrow\)TRL<<GRPO−0\.0146\-0\.0146−9\.73\\phantom\{0\}\-9\.73<10−20∗⁣∗∗\{<\}10^\{\-20\}\\,\{\}^\{\*\*\*\}TRL<<KL\-Reg−0\.0114\-0\.0114−10\.94\-10\.94<10−24∗⁣∗∗\{<\}10^\{\-24\}\\,\{\}^\{\*\*\*\}Distribution Overlap\(α\\alpha,↑\\uparrow\)TRL\>\>GRPO\+0\.0147\+0\.0147\+36\.16\+36\.16<10−139∗⁣∗∗\{<\}10^\{\-139\}\\,\{\}^\{\*\*\*\}TRL\>\>KL\-Reg\+0\.0110\+0\.0110\+30\.22\+30\.22<10−113∗⁣∗∗\{<\}10^\{\-113\}\\,\{\}^\{\*\*\*\}Table[3](https://arxiv.org/html/2606.28166#S5.T3)reports paired one\-sidedtt\-tests for the three directional claims involving TRL in the main text\. On handoff robustness \(§[4\.3](https://arxiv.org/html/2606.28166#S4.SS3)\), TRL’s lead over both GRPO and KL\-Reg holds at everykk, with all eight tests significant atp<0\.05p<0\.05and two tests atp<0\.001p<0\.001; the consistency acrosskkrules out a single\-kkartifact\. On junior legibility \(§[4\.5](https://arxiv.org/html/2606.28166#S4.SS5)\), TRL’s per\-token cross\-entropy is significantly lower than both baselines, confirming that the senior’s chain\-of\-thought is genuinely more predictable to the junior\. The distribution\-overlap gap is the sharpest: TRL beats both baselines atp<10−100p<10^\{\-100\}, providing essentially noise\-free support for the rollout\-driven anchoring described in §[4\.4](https://arxiv.org/html/2606.28166#S4.SS4)\.

## 6Conclusion

In this work, we introduced Tandem Reinforcement Learning \(TRL\), which carries the tandem training paradigm into the RLVR pipeline that drives current reasoning LLMs\. In TRL, each rollout is co\-generated by a trainable senior and a frozen junior initialized from the senior’s pre\-RL base, the two alternate at word boundaries, and the standard GRPO loss is applied to senior\-emitted tokens\. On competition math benchmarks with Qwen3\-4B\-Instruct, the TRL\-trained senior is no less capable than the GRPO\-trained one when reasoning alone\. Beyond this parity, the same rollout structure that enabled training also leaves measurable traces in the senior’s behavior at inference: its co\-generations with the junior succeed more often than GRPO’s, its token\-level distribution stays closer to the junior’s, and its reasoning steps are more readily predicted by the junior\.

## Future work

Several directions for future work are worth highlighting\. First, replacing the fixed self\-paired junior with a pool of diverse juniors varying in capability, style, language, or tool use could regularize the senior toward more broadly intelligible behavior\. Second, pairing the senior with juniors calibrated to specific human skill levels, analogous to Maia\(McIlroy\-Younget al\.,[2020](https://arxiv.org/html/2606.28166#bib.bib41)\)and Maia\-2\(Tanget al\.,[2024](https://arxiv.org/html/2606.28166#bib.bib42)\)for chess, would provide a more direct bridge to the genuine human\-AI handoff scenarios that originally motivated the tandem framework\. We view this direction as currently bottlenecked by the absence of comparably well\-calibrated, human\-skill\-level proxies for general\-purpose language models, and a promising avenue once such models mature\. Third, the mechanistic underpinnings of TRL warrant further study, including understanding the dynamics of TRL and why self\-pairing produces the three observed properties\.

## Acknowledgments

This research is funded by grants from the Natural Sciences and Engineering Research Council of Canada \(NSERC\), the Canada Foundation for Innovation, and the Ontario Research Fund\. We are grateful to Zhenwei Tang and Julian Minder for helpful discussions and feedback on this work\.

## References

- Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.\(2022\)Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p3.1),[§3\.1](https://arxiv.org/html/2606.28166#S3.SS1.p2.1)\.
- C\. Burns, P\. Izmailov, J\. H\. Kirchner, B\. Baker, L\. Gao, L\. Aschenbrenner, Y\. Chen, A\. Ecoffet, M\. Joglekar, J\. Leike,et al\.\(2023\)Weak\-to\-strong generalization: eliciting strong capabilities with weak supervision\.arXiv preprint arXiv:2312\.09390\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p2.1),[§4\.4](https://arxiv.org/html/2606.28166#S4.SS4.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§A\.2](https://arxiv.org/html/2606.28166#A1.SS2.p3.2),[§4\.1](https://arxiv.org/html/2606.28166#S4.SS1.p1.2),[§5\.3](https://arxiv.org/html/2606.28166#S5.SS3.p1.6)\.
- Gpg: a simple and strong reinforcement learning baseline for model reasoning\.arXiv preprint arXiv:2504\.02546\.Cited by:[§2](https://arxiv.org/html/2606.28166#S2.p2.1)\.
- Y\. Chu, B\. Wang, X\. Liu, H\. Chen, A\. Liu, and X\. Hu \(2025b\)Ssr: speculative parallel scaling reasoning in test\-time\.arXiv preprint arXiv:2505\.15340\.Cited by:[§4\.3](https://arxiv.org/html/2606.28166#S4.SS3.p1.1)\.
- T\. R\. Davidson, A\. Fourney, S\. Amershi, R\. West, E\. Horvitz, and E\. Kamar \(2025\)The collaboration gap\.arXiv preprint arXiv:2511\.02687\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p2.1),[§4\.3](https://arxiv.org/html/2606.28166#S4.SS3.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p1.1),[§1](https://arxiv.org/html/2606.28166#S1.p2.1),[§2](https://arxiv.org/html/2606.28166#S2.p2.1),[§2](https://arxiv.org/html/2606.28166#S2.p3.1),[§4\.4](https://arxiv.org/html/2606.28166#S4.SS4.p1.1)\.
- K\. Hamade, R\. McIlroy\-Young, S\. Sen, J\. Kleinberg, and A\. Anderson \(2024\)Designing skill\-compatible ai: methodologies and frameworks in chess\.arXiv preprint arXiv:2405\.05066\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p3.1),[§2](https://arxiv.org/html/2606.28166#S2.p1.1),[§3\.1](https://arxiv.org/html/2606.28166#S3.SS1.p1.2),[footnote 3](https://arxiv.org/html/2606.28166#footnote3)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.Note:NeurIPS 2021 Datasets and Benchmarks TrackExternal Links:2103\.03874,[Document](https://dx.doi.org/10.48550/arXiv.2103.03874),[Link](https://arxiv.org/abs/2103.03874)Cited by:[§A\.2](https://arxiv.org/html/2606.28166#A1.SS2.p1.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p3.1)\.
- A\. Karpov, T\. Adeleke, S\. H\. Cho, and N\. Perez\-Campanero \(2025\)The steganographic potentials of language models\.arXiv preprint arXiv:2505\.03439\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p2.1),[§2](https://arxiv.org/html/2606.28166#S2.p3.1),[§4\.4](https://arxiv.org/html/2606.28166#S4.SS4.p1.1)\.
- J\. H\. Kirchner, Y\. Chen, H\. Edwards, J\. Leike, N\. McAleese, and Y\. Burda \(2024\)Prover\-verifier games improve legibility of llm outputs\.arXiv preprint arXiv:2407\.13692\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p2.1),[§2](https://arxiv.org/html/2606.28166#S2.p3.1)\.
- S\. Kullback \(1997\)Information theory and statistics\.Courier Corporation\.Cited by:[§4\.4](https://arxiv.org/html/2606.28166#S4.SS4.p3.4)\.
- W\. Kwon \(2025\)VLLM: an efficient inference engine for large language models\.Ph\.D\. Thesis,UC Berkeley\.Cited by:[§A\.1](https://arxiv.org/html/2606.28166#A1.SS1.p1.1),[§5\.1](https://arxiv.org/html/2606.28166#S5.SS1.p3.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InInternational Conference on Machine Learning,pp\. 19274–19286\.Cited by:[§4\.5](https://arxiv.org/html/2606.28166#S4.SS5.p3.8)\.
- A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo,et al\.\(2022\)Solving quantitative reasoning problems with language models\.Advances in neural information processing systems35,pp\. 3843–3857\.Cited by:[§4\.1](https://arxiv.org/html/2606.28166#S4.SS1.p1.2)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu,et al\.\(2026\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.arXiv preprint arXiv:2604\.13016\.Cited by:[§2](https://arxiv.org/html/2606.28166#S2.p3.1)\.
- Y\. Li, J\. Xin, M\. M\. Miao, Q\. Long, and L\. Ungar \(2025\)The impact of language mixing on bilingual llm reasoning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 32519–32536\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p2.1),[§2](https://arxiv.org/html/2606.28166#S2.p3.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p2.1),[§1](https://arxiv.org/html/2606.28166#S1.p3.1),[§3\.1](https://arxiv.org/html/2606.28166#S3.SS1.p2.1),[§4\.4](https://arxiv.org/html/2606.28166#S4.SS4.p1.1)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025\)Understanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§2](https://arxiv.org/html/2606.28166#S2.p2.1)\.
- L\. C\. Magister, J\. Mallinson, J\. Adamek, E\. Malmi, and A\. Severyn \(2023\)Teaching small language models to reason\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 1773–1781\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p3.1)\.
- R\. McIlroy\-Young, S\. Sen, J\. Kleinberg, and A\. Anderson \(2020\)Aligning superhuman ai with human behavior: chess as a model system\.InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining,pp\. 1677–1687\.Cited by:[Future work](https://arxiv.org/html/2606.28166#Sx1.p1.1)\.
- H\. Meng, K\. Huang, S\. Wei, C\. Ma, S\. Yang, X\. Wang, G\. Wang, B\. Ding, and J\. Zhou \(2026\)Sparse but critical: a token\-level analysis of distributional shifts in rlvr fine\-tuning of llms\.arXiv preprint arXiv:2603\.22446\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p2.1),[§2](https://arxiv.org/html/2606.28166#S2.p3.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.Note:NeurIPS 2022External Links:2203\.02155,[Document](https://dx.doi.org/10.48550/arXiv.2203.02155),[Link](https://arxiv.org/abs/2203.02155)Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p3.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p1.1),[§2](https://arxiv.org/html/2606.28166#S2.p2.1),[§3\.1](https://arxiv.org/html/2606.28166#S3.SS1.p3.1),[§3\.4](https://arxiv.org/html/2606.28166#S3.SS4.p1.2),[§4\.1](https://arxiv.org/html/2606.28166#S4.SS1.p1.2)\.
- J\. Skaf, L\. Ibanez\-Lissen, R\. McCarthy, C\. Watts, V\. Georgiv, H\. Whittingham, L\. Gonzalez\-Manzano, D\. Lindner, C\. Tice, E\. J\. Young,et al\.\(2025\)Large language models can learn and generalize steganographic chain\-of\-thought under process supervision\.arXiv preprint arXiv:2506\.01926\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p2.1),[§2](https://arxiv.org/html/2606.28166#S2.p3.1),[§4\.4](https://arxiv.org/html/2606.28166#S4.SS4.p1.1)\.
- S\. Tan, M\. Luo, J\. Wong, C\. Cai, X\. Shi, W\. Y\. Tang, M\. Roongta, T\. Zhang, L\. E\. Li, R\. A\. Popa,et al\.\(2025\)DeepScaleR: effective rl scaling of reasoning models via iterative context lengthening\.Cited by:[§A\.2](https://arxiv.org/html/2606.28166#A1.SS2.p1.1),[§4\.1](https://arxiv.org/html/2606.28166#S4.SS1.p1.2)\.
- Z\. Tang, D\. Jiao, R\. McIlroy\-Young, J\. Kleinberg, S\. Sen, and A\. Anderson \(2024\)Maia\-2: a unified model for human\-ai alignment in chess\.Advances in Neural Information Processing Systems37,pp\. 20919–20944\.Cited by:[Future work](https://arxiv.org/html/2606.28166#Sx1.p1.1)\.
- J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins \(2022\)Solving math word problems with process\-and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p3.1)\.
- X\. Wang, J\. Wang, Y\. Wang, P\. Dang, S\. Cao, and C\. Zhang \(2025\)MARS: toward more efficient multi\-agent collaboration for llm reasoning\.arXiv preprint arXiv:2509\.20502\.Cited by:[§4\.3](https://arxiv.org/html/2606.28166#S4.SS3.p1.1)\.
- R\. West, A\. Anderson, E\. Kamar, and E\. Horvitz \(2026\)Tandem training for language models\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8265–8278\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p3.1),[§2](https://arxiv.org/html/2606.28166#S2.p1.1),[§3\.1](https://arxiv.org/html/2606.28166#S3.SS1.p1.2),[§3\.1](https://arxiv.org/html/2606.28166#S3.SS1.p2.1),[§3\.3](https://arxiv.org/html/2606.28166#S3.SS3.p1.13),[§5\.1](https://arxiv.org/html/2606.28166#S5.SS1.p4.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2606.28166#S4.SS1.p1.2)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu,et al\.\(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.External Links:2503\.14476,[Document](https://dx.doi.org/10.48550/arXiv.2503.14476),[Link](https://arxiv.org/abs/2503.14476)Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p1.1),[§2](https://arxiv.org/html/2606.28166#S2.p2.1),[§4\.4](https://arxiv.org/html/2606.28166#S4.SS4.p1.1)\.
- G\. Zeng, Z\. Zhou, D\. Arora, and A\. Zanette \(2025\)Shrinking the variance: shrinkage baselines for reinforcement learning with verifiable rewards\.arXiv preprint arXiv:2511\.03710\.Cited by:[§2](https://arxiv.org/html/2606.28166#S2.p2.1)\.
- R\. Zhan, Y\. Li, Z\. Wang, X\. Qu, D\. Liu, J\. Shao, D\. F\. Wong, and Y\. Cheng \(2025\)Exgrpo: learning to reason from experience\.arXiv preprint arXiv:2510\.02245\.Cited by:[§2](https://arxiv.org/html/2606.28166#S2.p2.1)\.
- Y\. Zhang, W\. Yao, C\. Yu, Y\. Liu, Q\. Yin, B\. Yin, H\. Yun, and L\. Li \(2025\)Improving sampling efficiency in rlvr through adaptive rollout and response reuse\.arXiv preprint arXiv:2509\.25808\.Cited by:[§2](https://arxiv.org/html/2606.28166#S2.p2.1)\.
- Y\. Zhao, Y\. Liu, J\. Liu, J\. Chen, X\. Wu, Y\. Hao, T\. Lv, S\. Huang, L\. Cui, Q\. Ye,et al\.\(2025\)Geometric\-mean policy optimization\.arXiv preprint arXiv:2507\.20673\.Cited by:[§2](https://arxiv.org/html/2606.28166#S2.p2.1)\.
- C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.\(2025a\)Group sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[§2](https://arxiv.org/html/2606.28166#S2.p2.1)\.
- H\. Zheng, Y\. Zhou, B\. R\. Bartoldson, B\. Kailkhura, F\. Lai, J\. Zhao, and B\. Chen \(2025b\)Act only when it pays: efficient reinforcement learning for llm reasoning via selective rollouts\.arXiv preprint arXiv:2506\.02177\.Cited by:[§2](https://arxiv.org/html/2606.28166#S2.p2.1)\.
- D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving \(2019\)Fine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[§1](https://arxiv.org/html/2606.28166#S1.p3.1)\.

## Appendix AReproducibility

### A\.1Tandem rollout implementation

A naive realization of tandem rollout couples two HuggingFace models in an outer Python loop with manual KV\-cache management\. We built such a prototype and found it impractical for RL training: at 512 generated tokens it exhausts the memory of a single 80 GB GPU, making RLVR under long chain\-of\-thoughts for mathematical reasoning tasks infeasible\. Moreover, per\-step throughput at shorter lengths is already roughly4×4\\timesslower than a single\-model vLLM\[Kwon,[2025](https://arxiv.org/html/2606.28166#bib.bib6)\]rollout\. Since RL training is dominated by rollout time, this slowdown alone makes large\-scale tandem RL infeasible\.

We therefore implement the tandem rollout*inside*vLLM, with bothπsen\\pi\_\{\\text\{sen\}\}andπjun\\pi\_\{\\text\{jun\}\}treated as paged\-attention\-aware models in the same engine\. The result is a decoding path that pays only the dual\-forward cost \(theoretically twice the single\-model latency\) and inherits every other vLLM optimization: paged KV cache, continuous batching, FlashAttention, CUDA graphs, and tensor parallelism\. Empirically our throughput is approximately0\.5×0\.5\\timesthat of a single\-model vLLM rollout, matching the architectural lower bound for sequential dual decoding\.

#### A\.1\.1Dual\-decoder backend

In our implementation,πsen\\pi\_\{\\text\{sen\}\}andπjun\\pi\_\{\\text\{jun\}\}are loaded onto separate devices and registered in the same vLLM engine, with the junior’s layers carrying a fixed name prefix so that the engine’s forward context can route them without interfering with the senior’s\. The two models share per\-step attention metadata \(slot mappings, block tables, position indices\), since they process identical token sequences, but each writes to its own KV\-cache tensor on its own device\. At each generation step the engine performs one forward pass throughπsen\\pi\_\{\\text\{sen\}\}, then one throughπjun\\pi\_\{\\text\{jun\}\}on the same appended token, samples independently from each model’s logits, and a tandem sampler emits the chosen token according to the active\-model schedule of §[3\.3](https://arxiv.org/html/2606.28166#S3.SS3); the chosen token is then fed to both models on the next step, keeping the two KV caches in lockstep with the shared responsey<ty\_\{<t\}\. This isolates the tandem\-specific logic to model loading, the second forward, and the sampler, while leaving every other vLLM runtime path untouched\.

For each request, the engine also accumulates a per\-token authorship stream alongside the generated tokens, flagging which positions were emitted byπsen\\pi\_\{\\text\{sen\}\}\. The stream is propagated through the same output channels as the tokens themselves \(model\-runner output, engine\-core output, per\-request completion output\) and is exposed to the verl trainer as a tensor field of the rollout batch\. On the trainer side, the actor’s loss path consumes it by elementwise\-multiplying it into the per\-token response mask before the policy\-gradient sum, which yields exactly the senior\-only loss of §[3\.4](https://arxiv.org/html/2606.28166#S3.SS4)\. This is the only addition to the verl GRPO surface; the actor, critic, and trainer code paths are otherwise unchanged\.

#### A\.1\.2Word\-level handoff details

With the vLLM\-native backend in place, we now detail the word\-level handoff schedule of §[3\.3](https://arxiv.org/html/2606.28166#S3.SS3)as instantiated in our implementation\.

The set of word\-boundary tokens is auto\-resolved from the active tokenizer at engine initialization\. For any BPE tokenizer using a leading\-space marker \(the Qwen3 / GPT\-style word\-initial prefix\), The boundary set is exactly the set of vocabulary IDs whose surface form begins with that marker\. For the Qwen3 tokenizer used in this work, this yields roughly53​k53\\text\{k\}of151​k151\\text\{k\}IDs\.

The boundary set alone, however, is not sufficient\. Mathematical reasoning produces long runs of non\-boundary tokens inside internal LaTeX expressions, variable identifiers, and code blocks, where leading\-space tokens never appear\. Without intervention, such runs would receive no handoff, allowing the active model to retain control across an entire derivation step and reintroducing exactly the reward\-hacking surface that word\-level granularity was meant to close\. We therefore add a backstop: if more thanKKconsecutive non\-boundary tokens have elapsed since the last handoff, the active model is redrawn under the same Bernoulli\(pp\) rule used at boundaries\. We useK=32K=32, chosen as a default that prevented monopoly behaviour in our early rollouts\.

The Bernoulli draws at boundary and cap\-fallback positions use the per\-request seeded random number generator that vLLM already maintains for sampling, so a given rollout’s authorship sequence is reproducible from its request seed\. To avoid replaying the full token history at every generation step, the tandem sampler maintains a small per\-request state \(active model, tokens since the last handoff, last\-seen response length\) and resumes from this state on each subsequent step\.

Table 4:Hyperparameters for TRL training on DeepScaleR with Qwen3\-4B\-Instruct\-2507\. The GRPO block is identical to the vanilla\-GRPO baseline; the Tandem block contains the TRL\-specific configurations\.

### A\.2Datasets

We train on DeepScaleR\[Tanet al\.,[2025](https://arxiv.org/html/2606.28166#bib.bib7)\], a corpus of approximately40,00040\{,\}000competition\-math problems compiled from past olympiad and contest sources\. Each example provides a problem statement and a verifiable final answer; reward is computed by extracting the final\\boxed\{\}expression from the model’s response and comparing it against the gold answer with a sympy\-based numeric or symbolic match adopted fromHendryckset al\.\[[2021](https://arxiv.org/html/2606.28166#bib.bib9)\]\.

For evaluation we use a panel of three competition\-math benchmarks, each disjoint from the training corpus:

- •AMC 23–25555Compiled from AMC 10/12 problems \(American Mathematics Competitions\) of 2023, 2024, and 2025\.: 121 problems\.
- •AIME 24–26666Compiled from AIME \(American Invitational Mathematics Examination\) of 2024, 2025, and 2026\.: 90 problems\.
- •

Each problem is formatted with an instruction requiring the model to output its final answer in\\boxed\{\}form\. We score every rollout with the same boxed\-extraction grader used during training; aggregate metrics are reported as pass@kkoverkkindependent rollouts per problem following the unbiased estimator proposed inChenet al\.\[[2021](https://arxiv.org/html/2606.28166#bib.bib8)\]\.

### A\.3Hyperparameter configuration

Table[4](https://arxiv.org/html/2606.28166#A1.T4)summarises the hyperparameters used for TRL training and vanilla GRPO baseline\. All TRL experiments reported in this paper were trained on 2×\\timesNVIDIA A100 80GB GPUs, withπsen\\pi\_\{\\text\{sen\}\}oncuda:0andπjun\\pi\_\{\\text\{jun\}\}oncuda:1; both models run with tensor\-parallel size11\.

## Appendix BAdditional Results

### B\.1Qualitative Analysis of Over\-emitted Tokens

Figure[6](https://arxiv.org/html/2606.28166#A2.F6)visualises the tokens each model over\-emits most relative to the junior, sized by drift magnitude\.

GRPO’s dominant over\-emitted tokens form two clusters\. The first is a pure enumeration artifact: the\-I,\-H,\-F,\-G,\-E,\-Cfamily, single\-letter labels that GRPO uses 71–146 times per corpus while the junior uses them 0–11 times\. These tokens carry no mathematical content: they are structural markers GRPO learned to impose on multi\-case reasoning, labelling each case with a capitalised suffix\. TRL recovers almost all of them to near\-junior levels \(1–30 occurrences\)\. The second cluster is reward\-induced answer signalling:Answergoes from 51 occurrences in the junior to 419 in GRPO \(8×\\times\), reflecting a learned habit of explicitly opening the answer statement; TRL reduces this to 116\.

TRL’s remaining over\-emitted tokens are qualitatively different\. The two largest aredom\(junior: 0, GRPO: 8, TRL: 103\) andct\(junior: 4, GRPO: 40, TRL: 104\), compact mathematical notation for*domain*and likely*cotangent*or a covariant transform, absent from the junior and barely present in GRPO\. Beyond these, TRL develops a coordinate\-subscript vocabulary that GRPO does not:\_l\(junior: 0, GRPO: 4, TRL: 55\),\-j\(junior: 5, GRPO: 3, TRL: 49\),\+x\(junior: 11, GRPO: 2, TRL: 53\),\+y\(junior: 9, GRPO: 39, TRL: 70\), thei,j,k,li,j,k,lcomponent\-labelling system of vector notation\.

The pattern is consistent with the structural account: GRPO’s most extreme over\-emitted tokens are formatting and signalling habits with no grounding in the mathematical content of the problems\. The co\-generation constraint eliminates these—they are precisely the choices the frozen junior cannot continue—while TRL’s residual drift is concentrated in compact mathematical notation that the junior’s vocabulary cannot recover by construction\.

![Refer to caption](https://arxiv.org/html/2606.28166v1/x6.png)Figure 6:Tokens most over\-emitted by Vanilla GRPO \(*left*\) and TRL \(*right*\) relative to the junior, sized by displacement magnitude\.
### B\.2Out\-of\-Distribution Junior Evaluation

#### B\.2\.1Handoff Robustness

The results in §[4\.3](https://arxiv.org/html/2606.28166#S4.SS3)concern only the training\-partner junior \(Qwen3\-4B\-Instruct\)\. As a secondary observation with no strong claims attached, we ask what happens when the same trained senior is paired with Qwen3\-family juniors of different sizes that were not seen during training\. These results are supplementary to our main framing and should not be read as evidence that the cooperation property generalises broadly; we include them as an empirical data point\.

Table[5](https://arxiv.org/html/2606.28166#A2.T5)reports pass@44for Qwen3\-0\.6B and Qwen3\-1\.7B juniors\. The most consistent pattern appears on AIME, where TRL leads by\+4\.5\+4\.5points with the 0\.6B junior and\+3\.5\+3\.5with the 1\.7B junior\. On AMC and Minerva the picture is more mixed: TRL leads with the weaker junior on AMC \(\+5\.8\+5\.8\) but falls marginally behind with the 1\.7B junior \(−0\.8\-0\.8\); Minerva differences are within1\.31\.3points in either direction\. We do not interpret this as a reliable generalisation signal\. The AIME pattern may reflect that on harder problems the alignment with the training partner provides a stronger lift, or it may be an artefact of sample size at these problem counts\. The one cell where GRPO leads \(Qwen3\-1\.7B, AMC\) is a useful reminder that the advantage is not monotone across partner sizes\.

Table 5:Reasoning\-step handoff robustness \(pass@44,↑\\uparrow\) with out\-of\-distribution juniors not seen during TRL training\.Boldmarks the higher value per cell;Δ=TRL−GRPO\\Delta=\\text\{TRL\}\-\\text\{GRPO\}\.
#### B\.2\.2Legibility with out\-of\-distribution juniors

We also pair each trained senior with Qwen3\-0\.6B and Qwen3\-1\.7B juniors it never encountered during training, and report legibility results alongside the in\-distribution partner in Table[6](https://arxiv.org/html/2606.28166#A2.T6)\. Interestingly, the in\-distribution legibility gain does not carry over to unseen juniors: with both smaller juniors, distribution overlap is matching or slightly lower for TRL than for GRPO, and cross\-entropy differences are mixed and small in magnitude\. However, this is a moderately expected property of a co\-generation objective that is, by design, specific to the training partner888Note that legibility is not translatable directly to handoff robustness\.\. We report these results as an initial characterization of how TRL legibility behaves under partner shift, and leave a systematic investigation to future work\.

Table 6:Legibility of senior CoTs across junior models of varying capability\. Qwen3\-4B\-Instruct is the in\-distribution training partner; the remaining rows are out\-of\-distribution\.

### B\.3Stochastic reasoning\-step handoff robustness

The main HR result in §[4\.3](https://arxiv.org/html/2606.28166#S4.SS3)alternates the senior and the junior strictly at every\\n\\nboundary\. This deterministic round\-robin is a clean evaluation protocol, but in scenarios such as agent\-to\-agent collaboration, drafter\-target inference, and human\-AI handoffs, the timing of handoffs is normally more flexible and the next contributor is typically chosen by the situation rather than by a fixed cadence\. As a robustness check, we replace the deterministic round\-robin with an independent Bernoulli\(p=0\.5p=0\.5\) draw at every\\n\\nboundary, keeping every other setting identical\. The expected senior share remains0\.50\.5, but the per\-rollout schedule now varies, and a team’s success has to be robust to that variation\.

Table 7:Reasoning\-step handoff robustness under stochastic Bernoulli \(p=0\.5p=0\.5\) schedules at\\n\\nboundaries \(pass@88, %,↑\\uparrow\)\. Junior is the Qwen3\-4B\-Instruct training partner\.Per\-rollout senior shares are well\-controlled across methods \(mean50\.0%50\.0\\%for both, standard deviation in the5%5\\%–9%9\\%range across benchmarks\)\. TRL retains a directional advantage over vanilla GRPO on every benchmark and on macro, with the largest gap again on AIME \(\+3\.3\+3\.3at pass@88\)\. The macro gap is smaller than under the strict schedule of §[4\.3](https://arxiv.org/html/2606.28166#S4.SS3)\(\+1\.8\+1\.8vs\.\+3\.1\+3\.1\), as expected: under a randomised schedule, individual rollouts vary in how much of the trajectory each model carries, and pass@88aggregates over schedule variation in addition to the usual generation variation\. The strict\-alternation protocol of the main text is the more stringent measurement; the stochastic protocol here verifies that TRL’s compatibility lift is not an artefact of a fixed schedule\.

## Appendix CWhy KL regularization toward the junior is structurally weaker than tandem rollouts

A natural ablation to TRL is to keep solo GRPO rollouts and add a per\-token KL penalty toward the frozen juniorπjun\\pi\_\{\\text\{jun\}\}\. This likewise “anchors” the senior toward the junior, but as a soft additive regularizer on the senior’s own trajectories rather than by sampling rollouts from a tandem mixture\. Empirically this baseline is strictly weaker than TRL on every anchoring axis we measure \(handoff robustness, marginal drift to base, and conditional legibility\)\. We give a structural account of why: the gradient of the KL\-Reg objective has access, at any regularization coefficientβ\\beta, to exactly two signals \(solo senior reward and per\-position divergence betweenπθ\\pi\_\{\\theta\}andπjun\\pi\_\{\\text\{jun\}\}\), neither of which is a function of whether the junior could continue fromhth\_\{t\}to a correct answer\. Sweepingβ\\betascales one of these two signals relative to the other, but does not introduce a third; the missing signal is supplied only by changing the rollout distribution, which is what TRL does\.

### C\.1Setup and what each objective optimizes

Let𝒱\\mathcal\{V\}be the vocabulary,xxa prompt,y=y1:T∈𝒱Ty=y\_\{1:T\}\\in\\mathcal\{V\}^\{T\}a response,ht=x⋅y<th\_\{t\}=x\\cdot y\_\{<t\}the context at positiontt, andr:𝒱T→\[0,1\]r:\\mathcal\{V\}^\{T\}\\to\[0,1\]a bounded verifier reward\. Bothπθ\\pi\_\{\\theta\}\(senior, trainable\) andπjun\\pi\_\{\\text\{jun\}\}\(junior, frozen\) are autoregressive:π​\(y∣x\)=∏t=1Tπ​\(yt∣ht\)\\pi\(y\\mid x\)=\\prod\_\{t=1\}^\{T\}\\pi\(y\_\{t\}\\mid h\_\{t\}\)\. We compare two objectives\.

##### \(a\) GRPO with KL regularization towardπjun\\pi\_\{\\text\{jun\}\}\.

ℒKL\-Reg\(θ\)=−𝔼y∼πθ\[∑tAtlogπθ\(yt∣ht\)\]\+β⋅𝔼y∼πθ\[∑tKL\(πθ\(⋅∣ht\)∥πjun\(⋅∣ht\)\)\]\.\\mathcal\{L\}\_\{\\text\{KL\-Reg\}\}\(\\theta\)\\;=\\;\-\\,\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\}\\\!\\Big\[\\textstyle\\sum\_\{t\}A\_\{t\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid h\_\{t\}\)\\Big\]\\;\+\\;\\beta\\cdot\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\}\\\!\\Big\[\\textstyle\\sum\_\{t\}\\mathrm\{KL\}\\\!\\big\(\\pi\_\{\\theta\}\(\\cdot\\mid h\_\{t\}\)\\,\\big\\\|\\,\\pi\_\{\\text\{jun\}\}\(\\cdot\\mid h\_\{t\}\)\\big\)\\Big\]\.\(2\)Rollouts are sampled solo fromπθ\\pi\_\{\\theta\}; the regularizer penalizes per\-position next\-token disagreement withπjun\\pi\_\{\\text\{jun\}\}along senior\-natural trajectories\.

##### \(b\) TRL\.

LetMpM\_\{p\}denote the tandem sampling: at each handoff boundary an independentBernoulli​\(p\)\\mathrm\{Bernoulli\}\(p\)draw determines the active modelat∈\{sen,jun\}a\_\{t\}\\in\\\{\\text\{sen\},\\text\{jun\}\\\}, andyt∼πat\(⋅∣ht\)y\_\{t\}\\sim\\pi\_\{a\_\{t\}\}\(\\cdot\\mid h\_\{t\}\)\.

ℒTRL​\(θ\)=−𝔼y∼Mp​\[∑t:at=senAt​log⁡πθ​\(yt∣ht\)\]\.\\mathcal\{L\}\_\{\\text\{TRL\}\}\(\\theta\)\\;=\\;\-\\,\\mathbb\{E\}\_\{y\\sim M\_\{p\}\}\\\!\\Big\[\\textstyle\\sum\_\{t\\,:\\,a\_\{t\}=\\text\{sen\}\}A\_\{t\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid h\_\{t\}\)\\Big\]\.\(3\)
Since only senior\-emitted positions depend onθ\\theta, the standard policy\-gradient identity gives

∇θ𝔼y∼Mp​\[r​\(y\)\]=𝔼y∼Mp​\[r​\(y\)​∑t:at=sen∇θlog⁡πθ​\(yt∣ht\)\]\.\\nabla\_\{\\theta\}\\,\\mathbb\{E\}\_\{y\\sim M\_\{p\}\}\[r\(y\)\]\\;=\\;\\mathbb\{E\}\_\{y\\sim M\_\{p\}\}\\\!\\Big\[r\(y\)\\textstyle\\sum\_\{t\\,:\\,a\_\{t\}=\\text\{sen\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid h\_\{t\}\)\\Big\]\.\(4\)Eq\. \([3](https://arxiv.org/html/2606.28166#A3.E3)\) with GRPO’s group\-relative advantage is thus a variance\-reduced unbiased gradient estimator of𝔼y∼Mp​\[r​\(y\)\]\\mathbb\{E\}\_\{y\\sim M\_\{p\}\}\[r\(y\)\]\. TRL directly optimizes the expected reward of joint rollouts sampled under the tandem mixture schedule\. Crucially,rrhere is evaluated on a trajectory that includes junior\-emitted tokens, so the team\-reward feedback at any senior\-emitted positionttdepends on whether the junior’s contributions are compatible with the rest of the trajectory\. The gradient on senior positionttis therefore reward\-weighted by a quantity that measures, for that position, whether the team carrying the junior’s tokens succeeds\. This is the signal that the KL\-Reg objective never receives\.

### C\.2KL\-Reg sees different signals than TRL at anyβ\\beta

The natural defense of KL\-Reg is to argue that the empirical gap to TRL is an unfortunate choice ofβ\\betaand can be closed by tuning\. We show it cannot: the KL\-Reg gradient is, at anyβ\\beta, a function of a strict subset of the signals the TRL gradient sees, and that subset omits exactly the quantity handoff success depends on\. No setting ofβ\\betarecovers the missing signal\.

From Eq\. \([2](https://arxiv.org/html/2606.28166#A3.E2)\), the gradient of the KL\-Reg objective decomposes as

∇θℒKL\-Reg​\(θ\)=−∇θ𝔼y∼πθ​\[∑tAt​log⁡πθ​\(yt∣ht\)\]⏟\(a\) GRPO on solo senior rollouts\+β⋅∇θ𝔼y∼πθ\[∑tKL\(πθ\(⋅∣ht\)∥πjun\(⋅∣ht\)\)\]⏟\(b\) per\-position divergence fromπjun\.\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{KL\-Reg\}\}\(\\theta\)\\;=\\;\\underbrace\{\-\\,\\nabla\_\{\\theta\}\\,\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\}\\\!\\Big\[\\textstyle\\sum\_\{t\}A\_\{t\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid h\_\{t\}\)\\Big\]\}\_\{\\text\{\(a\) GRPO on solo senior rollouts\}\}\\;\+\\newline \\beta\\cdot\\underbrace\{\\nabla\_\{\\theta\}\\,\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\}\\\!\\Big\[\\textstyle\\sum\_\{t\}\\mathrm\{KL\}\\\!\\big\(\\pi\_\{\\theta\}\(\\cdot\\mid h\_\{t\}\)\\,\\big\\\|\\,\\pi\_\{\\text\{jun\}\}\(\\cdot\\mid h\_\{t\}\)\\big\)\\Big\]\}\_\{\\text\{\(b\) per\-position divergence from $\\pi\_\{\\text\{jun\}\}$\}\}\.\(5\)Term \(a\) depends onr​\(y\)r\(y\)for trajectoriesy∼πθy\\sim\\pi\_\{\\theta\}; this is the senior’s solo reward, never evaluated on any trajectory in whichπjun\\pi\_\{\\text\{jun\}\}takes over\. Term \(b\) depends on the next\-token distributionsπθ\(⋅\|ht\)\\pi\_\{\\theta\}\(\\cdot\|h\_\{t\}\)andπjun\(⋅\|ht\)\\pi\_\{\\text\{jun\}\}\(\\cdot\|h\_\{t\}\)alone; it does not involverrat all\. Neither term is a function ofHtH\_\{t\}, the probability that the junior’s continuation fromhth\_\{t\}reaches a correct answer\. The coefficientβ\\betacontrols only the relative weight of \(a\) and \(b\); it cannot introduce the junior’s compatibility signal that is not present in either\.

Table[8](https://arxiv.org/html/2606.28166#A3.T8)summarises the resulting behaviour at each positionhth\_\{t\}, organised by the per\-position KL \(what KL\-Reg can see\) and by whether the junior could continue successfully \(what governs handoff success\)\. The two methods agree on the off\-diagonal and disagree on the diagonal; tuningβ\\betascales the row dimension without affecting the column\.

Table 8:What each method does at positionhth\_\{t\}, by per\-position KL betweenπθ\(⋅∣ht\)\\pi\_\{\\theta\}\(\\cdot\\mid h\_\{t\}\)andπjun\(⋅∣ht\)\\pi\_\{\\text\{jun\}\}\(\\cdot\\mid h\_\{t\}\)and by whether the junior could continue successfully \(HtH\_\{t\}\)\.
### C\.3Implications

##### Tuningβ\\betacannot rescue KL\-Reg on the axes the paper claims\.

A defender of the KL\-Reg baseline could in principle argue that the empirical gap to TRL on handoff robustness and conditional legibility reflects an unfortunate choice ofβ\\beta\. §[C\.2](https://arxiv.org/html/2606.28166#A3.SS2)closes this: raisingβ\\betacan only spend more pressure along the row axis of Table[8](https://arxiv.org/html/2606.28166#A3.T8), while every position in the bottom\-left cell remains invisible to the gradient\. Beyond some moderateβ\\beta, additional row\-axis pressure also pullsπθ\\pi\_\{\\theta\}towardπjun\\pi\_\{\\text\{jun\}\}on positions where the divergence carries the senior’s RLVR\-acquired capability, potentially degrading solo accuracy without recovering the column\-axis signal\.

##### Compute budgeting favors TRL\.

Polishing KL\-Reg training with carefully\-tunedβ\\betarequires sweepingβ\\beta, which multiplies the GRPO baseline cost linearly in the number of points swept\. Given the orthogonality argument, this sweep is structurally guaranteed either to fail to close the gap on the column\-axis\-dependent metrics, or to close them only by collapsing solo accuracy\. TRL has no analogous coefficient to sweep: the anchoring pressure is supplied by the rollout distribution rather than by a regularizer, and the only TRL\-specific hyperparameters \(ppandKK\) are determined by the granularity of the handoff schedule rather than by a tradeoff with solo capability\. The wall\-clock comparison in §5\.1 \(TRL 9\.4 hours, vanilla GRPO 7\.8 hours, under the best\-checkpoint protocol\) therefore understates TRL’s compute advantage relative to a fully defended KL\-Reg baseline, which would multiply the GRPO budget by the number ofβ\\betavalues swept\.

![Refer to caption](https://arxiv.org/html/2606.28166v1/x7.png)Figure 7:Vocabulary drift from base for GRPO, KL\-Reg, and TRL\.*Left*: survival curve of the absolute per\-token log\-ratio to the base over tokens\.*Right*: for the top\-500 most displaced tokens by GRPO, the per\-method drift relative to GRPO\.