Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

# Competitive Cross-Model RL with Implicit Rival Grading of Reasoning
Source: [https://arxiv.org/html/2607.07690](https://arxiv.org/html/2607.07690)
###### Abstract

Reinforcement learning from verifiable rewards \(e\.g\. GRPO\) is the engine behind today’s reasoning models, yet it grades only the final answer\. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists\. We introduceAgon, which makes two competing models each other’s graders\. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out\-solving the other\. To win, a model must out\-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model\. Because both models are optimized, each faces a progressively stronger rival, which single\-model RL cannot provide\. The two need only be comparably strong and behaviorally different\. At inference the pair deploys as it trains, a two\-stage cascade in which one model drafts and the other answers after reading the draft\. On the hard split of DeepMath with Qwen3, this doubles GRPO’s pass@1, roughly eight times the gain of an untrained Mixture\-of\-Agents pass over the same base\. The ordering replicates on competitive\-programming code and across model families \(Qwen3\.5, Gemma 4\)\. For now the models talk in text; the next step is to let them reason together in latent space\.

## 1Introduction

Zero\-shotGRPOMoACoopAgon0202040406060808023233030343446466161DeepMath\-hard pass@1 \(%\)Figure 1:From answer\-grading to rival\-grading\. On a hard\-math held\-out set, vanilla GRPO lifts a Qwen3 model only modestly over zero\-shot, and an untrained Mixture\-of\-Agents pass \(MoA, no train\), two models cross\-refining*at inference*, adds a small bump\. Training the same two\-model setup competitively \(Agon\) reaches roughly2×2\\timesGRPO’s and1\.8×1\.8\\timesthe untrained MoA’s pass@1 at the same two\-pass inference budget \([Table˜3](https://arxiv.org/html/2607.07690#S5.T3)\)\. Intermediate ablations appear in[Section˜5](https://arxiv.org/html/2607.07690#S5)\.Reinforcement learning from verifiable rewards has become the standard tool for sharpening the reasoning of large language models \(LLMs\) on tasks such as mathematics and code\(Guo et al\.,[2025](https://arxiv.org/html/2607.07690#bib.bib1); Shao et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib2)\)\. The dominant recipe is on\-policy and outcome\-based: sample a group of rollouts from the current policy, score each by whether its*final answer*is correct, and update with group\-relative advantages, as in GRPO\(Shao et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib2)\)\. The reward is attached to the answer; the chain of thought that produced it is never graded\. On problems the model can already partly solve this is benign, but on hard problems it creates a strong incentive to write more: a longer chain affords more chances to stumble onto the right answer, so additional text is the cheapest way to raise expected reward\. The trace inflates with hedging and backtracking \(“hmm,” “wait,” “let me reconsider”\), and accuracy grows far slower than length \(accuracy rises only modestly while length inflates by an order of magnitude\(Aggarwal & Welleck,[2025](https://arxiv.org/html/2607.07690#bib.bib9)\)\)\. The model ends up producing more reasoning per problem without producing better reasoning per token\.

The obvious fix, grading the reasoning directly, is intractable\. There is no ground truth for “good thinking”: no label says which step was the insight and which was filler, and a learned process reward model is expensive, brittle, and itself unverifiable\(Lightman et al\.,[2023](https://arxiv.org/html/2607.07690#bib.bib10)\)\. So the trace stays unscored, and the length pathology persists\. We ask instead:*can a second model supply the missing signal?*If a different policy attempts the same problem and we reward each model for*out\-reasoning the other*, then the trace is graded implicitly by the rival, without any process labels\. A step that leads to a win is reinforced; filler that the opponent exploits is punished\.

The grader must be a*different*model, and the game must be*competitive*\. Post\-training RL is, by construction, self\-improvement: the policy is optimized on signal it generates itself, which reinforces the very blind spots that created its errors: a model auditing its own work tends to plateau\(Huang et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib11)\)\. A second model with different failure modes breaks this closed loop\. But merely*having*two models is not enough\. Naively averaging or letting models agree, as in self\-consistency and mixture\-of\-agents\(Wang et al\.,[2023](https://arxiv.org/html/2607.07690#bib.bib12),[2024](https://arxiv.org/html/2607.07690#bib.bib13)\), tends to regress toward consensus and can*dilute*quality when the models differ in strength\(Li et al\.,[2025](https://arxiv.org/html/2607.07690#bib.bib14)\), while two near\-identical models gain little from one another\. The combination that works is a pair of comparable strength hypothesized to have different blind spots, placed under competitive pressure: each model must not only solve the problem but beat a peer that sees its work, so the objective rewards out\-reasoning the peer rather than agreeing with it\. Under this pressure the pair co\-improves: each win raises the bar for the other on the next step, and both models end up past the ceiling where a single self\-graded policy plateaus\.

We instantiate this asAgon\(Greek*agōn*, contest\)\. Two distinct policies are trained head\-to\-head: a*drafter*produces candidate solutions; a*challenger*reads an opponent’s worked solution and is rewarded for solving correctly*and*beating it; roles rotate every step so both models learn both roles, and at inference the pair deploys the same way, one model drafting and the other answering after reading the draft\. In principle the recipe applies to any pair of comparably strong models; for efficiency we realize the pair as two low\-rank adapters over one frozen base \(∼2%\{\\sim\}2\\%memory overhead, not2×2\\times\), kept divergent by different initialization and update streams\. In this work the exchange happens in text: we first establish that the cross\-model signal is useful before optimizing the channel \(latent\-space exchange;[Section˜6](https://arxiv.org/html/2607.07690#S6)\)\. We test two claims under a matched generation budget: that cross\-model information exchange raises accuracy \(the*exchange claim*\), and that adversarial pressure raises it further than cooperation \(the*competition claim*\), using a2×22\\times 2design that crosses competition with exchange \(one cell of which is provably inert under group\-relative normalization;[Section˜4\.4](https://arxiv.org/html/2607.07690#S4.SS4)\), plus a single\-model self\-refinement control, a zero\-training orchestration control, and a budget\-matched GRPO self\-cascade control\.

#### Contributions\.

- •We reframe*reasoning quality as an implicit reward*that a competing peer model can supply, turning the unlabeled “good thinking” problem into a head\-to\-head game played during RL \([Section˜4](https://arxiv.org/html/2607.07690#S4)\)\.
- •We propose Agon: a recipe for*jointly training two distinct policies*against each other with competitive GRPO, via draft\-and\-challenge rollouts, per\-step role rotation, and a competitive reward \(correctness plus a conversion bonus for out\-solving the opponent\)\. It runs on a stock GRPO trainer with no changes to the optimization core; a shared\-base dual\-adapter instantiation makes the pair nearly as cheap as a single model \([Section˜4](https://arxiv.org/html/2607.07690#S4)\)\.
- •We lay out a2×22\\times 2matrix \(competition×\\timesinformation exchange\) with self\-refinement and zero\-training controls \([Section˜5](https://arxiv.org/html/2607.07690#S5)\)\. One cell, competition without exchange, is inert by construction under group\-relative normalization \([Section˜4\.4](https://arxiv.org/html/2607.07690#S4.SS4)\), so the competition claim rests on the adversarial\-vs\-cooperative comparison; the fully isolated cell \(per\-rollout hidden drafts\) is left for future work\.
- •Under a matched generation budget on hard math, competition\+\+exchange beats cooperation, self\-refinement, and vanilla GRPO on pass@1, with the largest gains on the smallest models\. At the same doubled inference budget, the trained cascade nearly doubles the accuracy of an untrained Mixture\-of\-Agents pass over the same base, which itself barely improves on GRPO\.

## 2Related Work

#### Outcome\-only RL and the length pathology\.

GRPO\(Shao et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib2)\)and the R1 recipe\(Guo et al\.,[2025](https://arxiv.org/html/2607.07690#bib.bib1)\)reward only the verified final answer\. A well\-documented side effect is uncontrolled growth of trace length \(“overthinking”\), with accuracy rising far more slowly than tokens\(Aggarwal & Welleck,[2025](https://arxiv.org/html/2607.07690#bib.bib9); Chen et al\.,[2024a](https://arxiv.org/html/2607.07690#bib.bib15)\)\. Remedies act on the symptom: explicit length penalties or length\-controlled objectives\(Aggarwal & Welleck,[2025](https://arxiv.org/html/2607.07690#bib.bib9)\), and dynamic sampling that discards degenerate all\-correct/all\-wrong groups\(Yu et al\.,[2025](https://arxiv.org/html/2607.07690#bib.bib16)\)\. None grades the*content*of the trace, because step\-level labels do not exist and learned process reward models are costly and themselves unverifiable\(Lightman et al\.,[2023](https://arxiv.org/html/2607.07690#bib.bib10)\)\. Agon leaves the verifier untouched and instead derives a content signal from a competing peer\.

#### Self\-play and self\-improvement\.

A model can improve against versions of itself: SPIN\(Chen et al\.,[2024b](https://arxiv.org/html/2607.07690#bib.bib17)\)plays the current policy against its own past generations; Self\-Rewarding LMs\(Yuan et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib18)\)use the model as its own judge; and self\-play reasoners such as Absolute Zero\(Zhao et al\.,[2025](https://arxiv.org/html/2607.07690#bib.bib19)\)and R\-Zero\(Huang et al\.,[2025](https://arxiv.org/html/2607.07690#bib.bib20)\)co\-evolve a proposer and a solver instantiated from one model\. These methods are powerful but share a structural ceiling: a policy optimized on its own signal reinforces the blind spots that caused its errors, and self\-correction without external feedback is unreliable\(Huang et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib11)\)\. Agon differs by training*two distinct*policies with different initialization and update streams, so the grader is a genuinely different agent rather than a copy of the graded\. Multiagent finetuning\(Subramaniam et al\.,[2025](https://arxiv.org/html/2607.07690#bib.bib27)\)also diversifies several models from one base via distinct data streams, but toward cooperative ensembling; Agon maintains divergence for*adversarial grading*, where the distinct rival is what supplies the reward\.

#### Multi\-model collaboration, debate, and ensembles\.

Combining several models \(self\-consistency\(Wang et al\.,[2023](https://arxiv.org/html/2607.07690#bib.bib12)\), multi\-agent debate\(Du et al\.,[2023](https://arxiv.org/html/2607.07690#bib.bib21)\), and Mixture\-of\-Agents\(Wang et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib13)\)\) can improve robustness, but the aggregation is*cooperative*: it pushes toward consensus, which dilutes quality when members differ in strength\(Li et al\.,[2025](https://arxiv.org/html/2607.07690#bib.bib14)\), and yields little when they are near\-identical\. These are inference\-time procedures over frozen models\. Agon instead applies cross\-model pressure*during*RL and makes it competitive: a model is rewarded for out\-solving a peer that has seen its work, so the objective pays for beating the peer, not for matching it\. We use a cooperative variant only as an ablation to isolate the value of competition, and include a two\-agent MoA layer as a zero\-training control; it recovers roughly one\-eighth of Agon’s gain over GRPO \([Table˜3](https://arxiv.org/html/2607.07690#S5.T3)\)\. Closest in spirit is in\-context self\-refinement\(Madaan et al\.,[2023](https://arxiv.org/html/2607.07690#bib.bib22)\), where a model revises its*own*draft; Agon differs in that the draft comes from a*competing, distinct*peer and is selected by outcome under RL, not by self\-feedback\.

#### Adversarial and generator–verifier training\.

Discriminator/verifier signals have long shaped generators, from GANs\(Goodfellow et al\.,[2014](https://arxiv.org/html/2607.07690#bib.bib24)\)to prover–verifier games that improve the legibility of LLM solutions\(Kirchner et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib26)\)and adversarial games over language\(Cheng et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib23)\)\. Debate\(Irving et al\.,[2018](https://arxiv.org/html/2607.07690#bib.bib25)\)first framed a judged competition between two agents as a scalable proxy for supervising reasoning itself\. Agon is symmetric and rotational rather than role\-fixed: both models alternate between drafting and challenging, so neither degenerates into a pure critic, and both improve as solvers; the judge is a ground\-truth verifier rather than a learned or human one\. The eventual move from text to latent\-space exchange connects to work on inter\-model communication in representation space\(Beliaev,[2025](https://arxiv.org/html/2607.07690#bib.bib29)\), which we leave as future work\.

## 3Preliminaries: GRPO and the Unscored Trace

For a problemxxwith a verifierr​\(x,y\)∈\{0,1\}r\(x,y\)\\in\\\{0,1\\\}that checks only the final answer of a completiony=\(trace,answer\)y=\(\\text\{trace\},\\text\{answer\}\), GRPO\(Shao et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib2)\)samples a group ofGGrolloutsy1,…,yG∼πθ\(⋅∣x\)y\_\{1\},\\dots,y\_\{G\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)and forms the group\-relative advantage

Ai=r​\(x,yi\)−μσ,μ=1G​∑jr​\(x,yj\),σ=stdj​r​\(x,yj\),A\_\{i\}\\;=\\;\\frac\{r\(x,y\_\{i\}\)\-\\mu\}\{\\sigma\},\\qquad\\mu=\\tfrac\{1\}\{G\}\\textstyle\\sum\_\{j\}r\(x,y\_\{j\}\),\\quad\\sigma=\\mathrm\{std\}\_\{j\}\\,r\(x,y\_\{j\}\),\(1\)updatingπθ\\pi\_\{\\theta\}to raise the log\-probability of above\-average rollouts, with the usual KL regularization toward a reference \(the base\) omitted here for brevity; groups with zero reward variance \(all\-correct or all\-wrong, whereσ=0\\sigma=0\) carry no signal and are dropped\. The signal is entirely a function of the*answer*: two completions with the same answer receive the same advantage regardless of how they reasoned\.

#### The unscored trace\.

On hard problems the per\-problem solve rate𝔼y∼πθ​\[r​\(x,y\)\]\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\}\[r\(x,y\)\]is small, and the policy raises expected reward most cheaply by increasing the number of distinct attempts inside one trace: each restart, case split, or “let me reconsider” is another chance at the answer\. Because the trace is never scored, this padding is not penalized and is in fact reinforced through any correct rollout that contains it\. The result is the empirically observed regime where length grows much faster than accuracy\(Aggarwal & Welleck,[2025](https://arxiv.org/html/2607.07690#bib.bib9); Chen et al\.,[2024a](https://arxiv.org/html/2607.07690#bib.bib15)\); in our own runs, GRPO training grows the mean trace from the zero\-shot6\.16\.1k to8\.18\.1k tokens for\+7\+7pp accuracy \([Table˜3](https://arxiv.org/html/2607.07690#S5.T3)\)\. The policy increases the quantity of reasoning while its density, the useful signal per token, stays flat\. The bottleneck, then, is not quantity\. The prevailing axes of scale \(data, parameters, context, and sampled tokens\) all add more reasoning; Agon instead targets a different axis, signal per token, by making low\-density traces lose the competition\.

#### The missing grader\.

Closing this gap requires a reward that distinguishes a concise, sound trace from a long, lucky one, i\.e\. a signal on the reasoning itself\. Verifiers do not provide it, and process reward models require labels we do not have\(Lightman et al\.,[2023](https://arxiv.org/html/2607.07690#bib.bib10)\)\. Agon obtains the signal*relationally*: a peer policy attempts the samexx, reads the other’s worked solution \(the post\-reasoning summary, not the raw trace;[Section˜4\.3](https://arxiv.org/html/2607.07690#S4.SS3)\), and the reward depends on*which model out\-reasons the other*\. A trace whose flaws the opponent exploits \(a sign error, a dead end it avoids\) loses; a trace that leads to a win is reinforced\. The grader is thus a second model, and, because both models are optimized, the bar rises over training\. We formalize this next\.

## 4Method: Agon

Agon trains two policies,AAandBB, head\-to\-head on the same problems\. The design has four parts: a draft\-and\-challenge rollout that routes one model’s reasoning into the other’s context \([Section˜4\.3](https://arxiv.org/html/2607.07690#S4.SS3)\); a competitive reward that grades the challenger against the drafter \([Section˜4\.4](https://arxiv.org/html/2607.07690#S4.SS4)\); a compute\-parity accounting so all comparisons are fair \([Section˜4\.5](https://arxiv.org/html/2607.07690#S4.SS5)\); and an efficient instantiation of the two policies as divergent adapters over a shared base \([Section˜4\.2](https://arxiv.org/html/2607.07690#S4.SS2)\)\.

problemxxdrafterAANNrollouts\{ai\}\\\{a\_\{i\}\\\}own GRPO updatechallengerBBNNrollouts\{bi\}\\\{b\_\{i\}\\\}trainablerewardR​\(bi\)R\(b\_\{i\}\)correctness\+\+conversionGRPO updatesBB: competitiveAA: standalonepeek:AA’s summary\(answer withheld\)aia\_\{i\}outcome \(for conversion\)roles rotate each stepFigure 2:One Agon step\. The drafterAAproducesNNsolutions from the plain prompt and receives a vanilla GRPO update on them \(its standalone stream\); the challengerBBreads each opponent’s*post\-reasoning solution summary*in\-context \(final answer withheld\), produces one paired rollout per opponent \(NNtotal\), and receives the competitive reward, which uses the paired draft’s outcome for the conversion bonus\. Both adapters are updated every step\. Roles rotate every optimizer step, so both models train in both streams\.AAandBBare two LoRA adapters over one frozen base\.### 4\.1Why a second model

Post\-training RL is self\-improvement: the gradient in[Equation˜1](https://arxiv.org/html/2607.07690#S3.E1)is computed from rollouts the policy itself produced, so it amplifies whatever the policy already does, including the systematic errors that define its blind spots\. A model checking its own work inherits the bias that produced the mistake, which is why unaided self\-correction often fails to add signal\(Huang et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib11)\); we expect the same closed loop to cap self\-play, however many rounds it runs \(illustrated in[Figure˜3](https://arxiv.org/html/2607.07690#S4.F3), left\)\. A*different*model is outside this loop: a peer hypothesized to have different failure modes audits what the policy cannot see in itself\. In principle the peer could come from a different model family: the strongest models, open ones included, now cluster within a few points of one another on standard reasoning benchmarks\(Stanford HAI,[2026](https://arxiv.org/html/2607.07690#bib.bib28)\), so pairs of matched strength are easy to find; whether such pairs also differ enough in behavior is an empirical question we leave to future work\. Our instantiation instead engineers the divergence structurally from one base \([Section˜4\.2](https://arxiv.org/html/2607.07690#S4.SS2)\)\.

Pairing helps only under two conditions\. The models must be of*comparable strength*\(otherwise, we expect, the weaker simply imitates the stronger and the game collapses into distillation; we do not ablate the strength gap\), and they must have*different failure modes*, so that one can catch what the other misses \(we make this concrete in[Section˜4\.2](https://arxiv.org/html/2607.07690#S4.SS2),[Figure˜4](https://arxiv.org/html/2607.07690#S4.F4)\)\. We therefore keep the pair matched in capacity but divergent in behavior, and we make the interaction*competitive*so the objective rewards beating the peer rather than agreeing with it\.

self\-play: closed loopAgon: open loopvsAABaudits itself with its own bias→\\toplateaugraded by different blind spots→\\tothe grader improves tooFigure 3:Why a second model\.Left:a single policy optimized on its own rollouts re\-audits with the same biases that caused its errors, so self\-play saturates\(Huang et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib11)\)\.Right:Agon pairs two divergent policies; each grades the other with different blind spots, each is rewarded for winning, and because both are optimized, the grader improves with the graded\.
### 4\.2Instantiating the two policies

Agon itself only requires two policies of matched strength and different blind spots; how they are parameterized is an implementation choice\. We instantiate both as LoRA adapters\(Hu et al\.,[2022](https://arxiv.org/html/2607.07690#bib.bib8)\)over a single frozen base model:πA=base\+ΔA\\pi\_\{A\}=\\mathrm\{base\}\\\!\+\\\!\\Delta\_\{A\}andπB=base\+ΔB\\pi\_\{B\}=\\mathrm\{base\}\\\!\+\\\!\\Delta\_\{B\}\. The second policy costs one extra rank\-16 adapter \(∼10\{\\sim\}10M parameters,∼2%\{\\sim\}2\\%of the base, plus its optimizer state\) rather than a full second model, and the shared base keeps the two policies in compatible representation spaces, useful for the eventual latent\-space exchange \([Section˜6](https://arxiv.org/html/2607.07690#S6)\)\. Two models alone are not enough, though: the useful regime is the one motivated in[Section˜4\.1](https://arxiv.org/html/2607.07690#S4.SS1),*equal strength with hypothesized different blind spots*\([Figure˜4](https://arxiv.org/html/2607.07690#S4.F4)\)\. We maintain divergence by \(i\) different initialization \(BBstarts with small Gaussian noise around the zero matrix;AAuses the standard LoRA zero\-init\) and \(ii\) different update streams induced by role rotation \([Section˜4\.3](https://arxiv.org/html/2607.07690#S4.SS3)\): on any given problem one adapter drafts while the other challenges, so the two never receive the same gradient\. All reported numbers use the same base checkpoint for both adapters\. We stress that this instantiation is*minimally*divergent: at initialization the two policies differ only by adapter noise, so any complementarity must emerge from the diverging update streams during training rather than from pre\-existing blind\-spot differences\. Different blind spots are therefore a hypothesis about the trained pair, not a measured property\. The gap between cooperative exchange and single\-adapter self\-refinement \(46 vs 32;[Table˜3](https://arxiv.org/html/2607.07690#S5.T3)\) shows that training to read a*peer’s*draft beats training to read one’s own, but the comparison varies more than one factor \(trainable adapter capacity, whose draft is read, the refiner prompt\), so it does not isolate blind\-spot divergence as the cause\.

clones \(same model, diff\. temperature\)overlap×\\times×\\times×\\timessame blind spots→\\tocorrelated failuresvsdesigned divergenceAB×\\timesdifferent blind spots→\\tocomplementary errorsFigure 4:Designed divergence \(schematic of the target regime, not a measurement\)\. Two copies of one model \(left\) cover the same problems and fail on the same ones, so a second copy adds little\. A pair that is equally strong but behaviorally different \(right\) has complementary error profiles, which is what makes cross\-model grading pay off\. Our dual\-adapter instantiation starts near the left picture \(same base, noise\-level adapter difference\) and relies on divergent initialization and rotated update streams to move toward the right during training; we do not measure how far it gets\.
### 4\.3The draft\-and\-challenge step

Each optimizer step \([Figure˜2](https://arxiv.org/html/2607.07690#S4.F2),[Algorithm˜1](https://arxiv.org/html/2607.07690#alg1)\) designates a*drafter*and a*challenger*\. The drafter generatesNNrollouts\{ai\}∼πdraft\(⋅∣x\)\\\{a\_\{i\}\\\}\\sim\\pi\_\{\\text\{draft\}\}\(\\cdot\\mid x\)from the plain problem prompt and receives a vanilla GRPO update on them \(correctness and format only, no opponent term\); this standalone stream trains each adapter to solve from scratch, which is the first stage of deployment \([Section˜4\.5](https://arxiv.org/html/2607.07690#S4.SS5)\)\. The challenger then conditions on its paired opponent’s*solution summary*: the answer section the model writes after its private reasoning block, which recaps the derivation\. The raw thinking trace is dropped, since the summary carries the distilled solution at a fraction of the tokens \(cheaper to prefill, and free of the exploration noise that dominates long traces\), and the*final answer is withheld*as a precaution against copying \([Figure˜5](https://arxiv.org/html/2607.07690#S4.F5)\), though a worked summary often implies its answer anyway\. The summary is inserted as an attempt that “may or may not be correct,” and the challenger generates one rollout per pair,bi∼πchal\(⋅∣x,ai\)b\_\{i\}\\sim\\pi\_\{\\text\{chal\}\}\(\\cdot\\mid x,a\_\{i\}\), forming a group ofNN\. The challenger receives the competitive gradient; each stream’s GRPO advantage is computed within its own group ofNN\. Because eachbib\_\{i\}sees a different opponentaia\_\{i\}, the group spans varied opponent contexts for the samexx; this is what gives the conversion bonus within\-group variance\.Roles rotate every step: on even stepsAAdrafts andBBchallenges, on odd steps the reverse\. Without rotation, each adapter receives only one kind of gradient, one only ever solving from scratch and the other only ever solving against an opponent; the pair still trains, but ends up weaker \(52 vs 61 pass@1;[Table˜7](https://arxiv.org/html/2607.07690#S5.T7)\)\. With rotation, both adapters alternate between the two streams every step and co\-improve\. The challenger prompt instructs the model not to trust or paraphrase the opponent but to verify it and solve independently, winning by being correct when the opponent is wrong; the prompt is identical across the coop and adv reward stacks \([Section˜4\.4](https://arxiv.org/html/2607.07690#S4.SS4)\), so they differ only in the reward\.

AreasoningBreasoningeach peeks at the rival’s summaryspots a sign error in eq\. 3skips a failed hypothesisborrows the angle symmetryAA’s answerBB’s answerwho out\-scores?reward==correctness\+\+bonus for out\-solving the rivalFigure 5:Peek and outscore\. A conceptual view aggregated over one rotation cycle: within a single step the exchange is one\-directional \([Figure˜2](https://arxiv.org/html/2607.07690#S4.F2)\); roles swap on the next step, so each model both reads and is read, on different problems\. While solving, the challenger reads the rival’s*worked solution*\(the summary it writes after its private reasoning, final answer withheld\) and writes against it, catching its errors \(a sign error in eq\. 3\), pruning its dead ends, and borrowing its good moves; the annotations are what the reader of the summary exploits, not messages exchanged between the models\. Neither solution is labeled, yet by rewarding whoever out\-solves the other, the rival’s reasoning is graded implicitly, a signal that outcome\-only RL does not use\.A \(drafter\) <think\>Okay, so I need to compute this integral\. Setu=1−x2u=1\-x^\{2\}, sod​u=2​x​d​xdu=2x\\,dx\. Wait, that’s negative? Hmm, that can’t be right\. Maybe I made a mistake\. Let me try integration by parts instead…*\(continues for∼1\.9\{\\sim\}1\.9k tokens; never shown toBB\)*</think\> summary*\(whatBBsees\)*:Substitutingu=1−x2u=1\-x^\{2\}withd​u=2​x​d​xdu=2x\\,dx\(sign dropped:d​u=−2​x​d​xdu=\-2x\\,dx\)gives∫01x​1−x2​𝑑x=12​∫10u​𝑑u\\int\_\{0\}^\{1\}x\\sqrt\{1\-x^\{2\}\}\\,dx=\\tfrac\{1\}\{2\}\\int\_\{1\}^\{0\}\\\!\\sqrt\{u\}\\,du, a negative value…*final answer withheld fromBB*B \(challenger\), readsAA’s summary <think\>The attempt’s substitution is right but the sign ofd​uduwas dropped:x​d​x=−12​d​ux\\,dx=\-\\tfrac\{1\}\{2\}\\,du, which flips the limits back and gives\+13\+\\tfrac\{1\}\{3\}\. The area is positive, no need to re\-explore\. Check at the boundary: integrand≥0\\geq 0on\[0,1\]\[0,1\]\.</think\> <answer\>1/31/3</answer\>correct,∼4×\{\\sim\}4\\timesshorter

Figure 6:Illustrative exchange \(hand\-constructed for clarity\)\. The drafter drops a sign, distrusts its own \(correct\-magnitude\) result, and spirals into re\-exploration; its post\-reasoning summary recaps the flawed derivation \(but does not contain the boxed answer\)\. The challenger sees*only*that summary \(raw thinking trace dropped, final answer withheld as a precaution\), pinpoints the flawed step, re\-derives it, and answers correctly in a fraction of the tokens\. The solution summary carries the derivation, which is what the challenger exploits\.
### 4\.4Reward: correctness plus a conversion bonus

Letc​\(y\)=r​\(x,y\)∈\{0,1\}c\(y\)=r\(x,y\)\\in\\\{0,1\\\}be correctness\. We report two reward stacks, applied to the challenger rolloutbib\_\{i\}given its paired drafter solutionaia\_\{i\}:

coop:R​\(bi\)=2​c​\(bi\)\+λ​ϕ​\(bi\),\\displaystyle R\(b\_\{i\}\)\\;=\\;2\\,c\(b\_\{i\}\)\\;\+\\;\\lambda\\,\\phi\(b\_\{i\}\),\(2\)adv:R​\(bi\)=2​c​\(bi\)\+c​\(bi\)​\(1−c​\(ai\)\)⏟conversion bonus\+λ​ϕ​\(bi\),\\displaystyle R\(b\_\{i\}\)\\;=\\;2\\,c\(b\_\{i\}\)\\;\+\\;\\underbrace\{c\(b\_\{i\}\)\\,\\big\(1\-c\(a\_\{i\}\)\\big\)\}\_\{\\text\{conversion bonus\}\}\\;\+\\;\\lambda\\,\\phi\(b\_\{i\}\),\(3\)whereϕ​\(⋅\)\\phi\(\\cdot\)is a format term \(well\-formed<think\>/<answer\>structure\) with weightλ=0\.5\\lambda=0\.5\(25% of the correctness term\), identical across stacks so it introduces no bias between them\. Incoopthe opponent appears only in the*context*, not in the reward, which isolates the value of information exchange \(the exchange claim\)\. Inadvthe*conversion bonus*c​\(bi\)​\(1−c​\(ai\)\)c\(b\_\{i\}\)\\,\(1\-c\(a\_\{i\}\)\)raises the effective correctness weight from22to33exactly on the paired opponent attempts that*failed*: solving where the opponent could not is worth more, which is the competitive pressure\.

The form matters\. A naïve marginc​\(bi\)−c​\(ai\)c\(b\_\{i\}\)\-c\(a\_\{i\}\)works poorly: sincec​\(ai\)c\(a\_\{i\}\)is*action\-independent*\(fixed by the pairingaia\_\{i\}, not by the challenger’s rolloutbib\_\{i\}\), its subtractive term contributes*no expected policy gradient*\(exactly in the unnormalized objective; up to the nonlinearity of group standardization after it\): it acts as a per\-sample baseline, shifting each reward by the drafter’s outcome without adding any competitive direction\. The tested margin stack,R=2​c​\(bi\)\+\(c​\(bi\)−c​\(ai\)\)=3​c​\(bi\)−c​\(ai\)R=2c\(b\_\{i\}\)\+\\big\(c\(b\_\{i\}\)\-c\(a\_\{i\}\)\\big\)=3c\(b\_\{i\}\)\-c\(a\_\{i\}\), makes this concrete: the scalar33cancels under group\-standard\-deviation normalization and the−c​\(ai\)\-c\(a\_\{i\}\)term is action\-independent \(zero expected policy gradient\), so in expectation the margin points in the*same direction*as the cooperative gradient\. The ablation is consistent with this prediction \(4949vs coop4646\), though the33pp difference lies within binomial sampling noise\. The conversion bonus instead*modulates the correctness weight by opponent difficulty*: because it multiplies the action\-dependentc​\(bi\)c\(b\_\{i\}\), the reweighting shifts the gradient itself, and, given the within\-group spread of opponent contexts\{ai\}\\\{a\_\{i\}\\\}, tilts it toward the opponent attempts that failed\. We confirm this empirically \([Table˜7](https://arxiv.org/html/2607.07690#S5.T7)\): the margin form barely improves on cooperation, whereas the conversion bonus clearly does\. More generally, any reward that is affine inc​\(bi\)c\(b\_\{i\}\)with group\-constant coefficients \(e\.g\., a shared opponent draft for the entire challenger group, makingc​\(a\)c\(a\)identical for allNNrollouts\) normalizes to exactly the advantage of the conversion\-free reward in the same contexts under group\-relative standardization; the conversion term then contributes zero expected policy gradient\. Within\-group variation in opponent difficulty is what makes the competitive term trainable\. In the shared\-opponent ablation we therefore fix one opponent draft per problem,*hidden*from the challenger’s context and used only for the reward \(so the contexts are plain prompts, as in vanilla GRPO\); the observed result \(32\) is consistent with the theoretical prediction that a group\-constant conversion bonus adds no directional signal\. This departs from the standard GRPO assumption that all rollouts in a group share the same prompt; here each challenger rollout conditions on its own opponent context\. The resulting credit assignment is not merely noisier but systematically confounded: a rollout’s success probability depends on the quality of the draft it happened to draw \(an action\-independent draw\), so group standardization mixes context difficulty with policy quality, and the conversion bonus, which pays exactly in the harder contexts, partially offsets that difficulty penalty\. This admits an alternative reading of the adversarial gain: rather than “competitive pressure” in a game\-theoretic sense, the bonus acts as difficulty\-weighted reward shaping that upweights correct completions in hard contexts\. Our experiments do not distinguish these two interpretations; we use the competitive framing for its constructive role in the design and leave the disentanglement to future work\.

The resulting reward ladder \(adv\) isR−λ​ϕ∈\{0,2,3\}R\-\\lambda\\phi\\in\\\{0,2,3\\\}\. The drafter’s standalone stream has no opponent, so[Equation˜2](https://arxiv.org/html/2607.07690#S4.E2)applies as\-is\.

As an*optional density lever*, studied separately in[Section˜5\.6](https://arxiv.org/html/2607.07690#S5.SS6)and*not*part of the headline recipe, we add a length tiebreak that fires only when both solutions are correct:R​\(bi\)\+⁣=γ​c​\(bi\)​c​\(ai\)​1​\[\|bi\|<\|ai\|\]R\(b\_\{i\}\)\\mathrel\{\+\{=\}\}\\gamma\\,c\(b\_\{i\}\)\\,c\(a\_\{i\}\)\\,\\mathbf\{1\}\\\!\\left\[\\,\|b\_\{i\}\|<\|a\_\{i\}\|\\,\\right\]with weightγ\\gamma, rewarding the challenger for matching the opponent’s correctness in*fewer*tokens\. Because the term requiresc​\(bi\)=1c\(b\_\{i\}\)=1, it never*directly*penalizes correctness, only breaking ties among already\-correct traces, so accuracy stays essentially flat while length drops, converting the emergent shortening \([Section˜5\.5](https://arxiv.org/html/2607.07690#S5.SS5)\) into a directly tunable target\.

Algorithm 1Agon \(one optimizer step\)1:input:batch of problems

\{x\}\\\{x\\\}; adapters

A,BA,B; group size

NN; step index

tt; mode

∈\{coop,adv\}\\in\\\{\\text\{coop\},\\text\{adv\}\\\}
2:

\(draft,chal\)←\(A,B\)\(\\text\{draft\},\\text\{chal\}\)\\leftarrow\(A,B\)if

tteven else

\(B,A\)\(B,A\)// role rotation

3:for all

xxin batchdo

4:

\{ai\}i=1N∼πdraft\(⋅∣x\)\\\{a\_\{i\}\\\}\_\{i=1\}^\{N\}\\sim\\pi\_\{\\text\{draft\}\}\(\\cdot\\mid x\)// standalone rollouts, plain prompt

5:

R​\(ai\)←R\(a\_\{i\}\)\\leftarrowreward via[Equation˜2](https://arxiv.org/html/2607.07690#S4.E2)// correctness\+\+format, no opponent term

6:

\{bi\}i=1N∼πchal\(⋅∣x,ai\)\\\{b\_\{i\}\\\}\_\{i=1\}^\{N\}\\sim\\pi\_\{\\text\{chal\}\}\(\\cdot\\mid x,a\_\{i\}\)// one per pair: challengerbib\_\{i\}seesaia\_\{i\}’s summary

7:

R​\(bi\)←R\(b\_\{i\}\)\\leftarrowreward via[Equation˜2](https://arxiv.org/html/2607.07690#S4.E2)/[Equation˜3](https://arxiv.org/html/2607.07690#S4.E3)by mode

8:group\-relative advantages of

\{R​\(ai\)\}\\\{R\(a\_\{i\}\)\\\}and of

\{R​\(bi\)\}\\\{R\(b\_\{i\}\)\\\}//[Equation˜1](https://arxiv.org/html/2607.07690#S3.E1), per group

9:endfor

10:GRPO update on

πdraft\\pi\_\{\\text\{draft\}\}\(standalone stream\) and on

πchal\\pi\_\{\\text\{chal\}\}\(competitive stream\)

11:returnupdated adapters

### 4\.5Compute parity

Per step Agon generatesNNdrafter rollouts plusNNchallenger rollouts \(2​N2N\), versusNNfor vanilla GRPO\. Both adapters are updated every step, the drafter on its standalone group and the challenger on its opponent\-conditioned group; rotation swaps which adapter gets which stream\. Our primary invariant is the*generation budget*: every method generates2​N2Nrollouts per training problem \(vanilla GRPO gets2​N2Ninstead of its defaultNN\) under the same completion\-length cap and the same single pass over the training problems\. Realized token counts then differ only through learned trace length, bounded for every method by the same cap; we do not audit total generated tokens \(the drafter\-stage length is not reported\), but with the rollout count and length cap equalized, the gains are not attributable to a larger generation*budget*\. The challenger additionally prefills the opponent summary, which is extra prefill compute we do not equalize \(the summary is short relative to a full trace, is amortized cheaply against generation, and is absent at single\-adapter inference\)\. At inference Agon deploys exactly as it trains: a two\-stage cascade in which one adapter drafts from the plain prompt and the other reads the draft’s summary and produces the final answer\. This is two sequential generation passes, a doubled budget relative to a single\-pass baseline, so[Table˜3](https://arxiv.org/html/2607.07690#S5.T3)includes two\-pass controls at the same budget \(cross\-refinement, self\-refinement\)\. Reported trace lengths refer to the final \(challenger\) stage\. Both cascade directions \(A→BA\{\\to\}BandB→AB\{\\to\}A\) are evaluated and the better is reported, selected post hoc on the held\-out set\.

## 5Experiments

#### Setup\.

We train Qwen3 models\(Qwen Team,[2025](https://arxiv.org/html/2607.07690#bib.bib3)\)\(primarily 0\.6B; 1\.7B/4B for scaling\) on the hard split of DeepMath\-103K\(He et al\.,[2025](https://arxiv.org/html/2607.07690#bib.bib7)\)\(difficulty 8, binary\-answer questions removed\), with a held\-out set of300300problems\. We use group sizeG=N=8G=N=8for both streams, LoRA rank1616, learning rate5×10−55\\times 10^\{\-5\}\(linear decay,10%10\\%warmup\), one training epoch over the3,0003\{,\}000training problems, format weightλ=0\.5\\lambda=0\.5, length tiebreakγ=0\.5\\gamma=0\.5\(used in[Section˜5\.6](https://arxiv.org/html/2607.07690#S5.SS6)only\), and a standard format reward \(well\-formed<think\>/<answer\>\) asϕ\\phi\. Generation uses vLLM\(Kwon et al\.,[2023](https://arxiv.org/html/2607.07690#bib.bib31)\)and training uses TRL\(von Werra et al\.,[2020](https://arxiv.org/html/2607.07690#bib.bib30)\)\. The primary metric is pass@1 on the DeepMath held\-out set; we additionally sanity\-check on GSM8K\(Cobbe et al\.,[2021](https://arxiv.org/html/2607.07690#bib.bib33)\)and MATH\-500\(Hendrycks et al\.,[2021](https://arxiv.org/html/2607.07690#bib.bib32)\)\([Table˜2](https://arxiv.org/html/2607.07690#S5.T2)\)\. For the second\-domain study \([Section˜5\.3](https://arxiv.org/html/2607.07690#S5.SS3)\) we train on theeasy\-difficulty subset of CodeContests\(Li et al\.,[2022](https://arxiv.org/html/2607.07690#bib.bib6)\)with a unit\-test verifier \(r=1r=1iff all tests pass\), under the same protocol and budget\. Evaluation protocol: sampled decoding \(temperature0\.60\.6, top\-pp0\.950\.95\), one rollout per stage, generation budget matched to training \(1515k completion tokens per stage\), the answer parsed from the<answer\>block; the300300held\-out problems are difficulty\-8 only with binary\-answer questions removed, split with a fixed seed before any training\. Pair\-trained methods are evaluated as the two\-stage cascade \([Section˜4\.5](https://arxiv.org/html/2607.07690#S4.SS5)\) in both directions, reporting the better direction; mean trace length reports the final\-stage completion\. Reported pass@1 values include 95% Clopper–Pearson binomial confidence intervals over the 300\-problem held\-out set \([Table˜4](https://arxiv.org/html/2607.07690#S5.T4)\)\.

#### The2×22\\times 2design\.

We cross two binary factors,*competition*and*information exchange*, giving four cells plus three controls \([Table˜1](https://arxiv.org/html/2607.07690#S5.T1)\): self\-refinement, an untrained MoA at inference, and a budget\-matched GRPO two\-pass self\-cascade\. One caveat is structural: the “competition without exchange” cell uses a shared opponent draft, hidden from the challenger and used only for the reward, which by the analysis in[Section˜4\.4](https://arxiv.org/html/2607.07690#S4.SS4)makes the conversion bonus group\-constant and hence gradient\-free, so this cell is theoretically equivalent to vanilla GRPO and serves as a consistency check rather than a true isolation of competition\. Consequently, the*exchange claim*\(exchange helps\) is carried by the GRPO\-vs\-cooperative comparison, and the*competition claim*\(competition helps beyond exchange\) by the cooperative\-vs\-adversarial comparison, holding everything else fixed at a matched generation budget; a competition\-without\-visible\-exchange cell with per\-rollout hidden drafts is left for future work\.

Table 1:The ablation matrix\. Each cell is a training configuration; everything but the two factors is held fixed\. Self\-refinement is a single\-model control for exchange; the untrained MoA at inference is a zero\-training Mixture\-of\-Agents control\.Controls: self\-refinement \(model sees its*own*prior attempt\); untrained MoA at inference \(the base model’s two*independent*samples critique each other, no training\); GRPO two\-pass self\-cascade \(a trained GRPO model refines its own first pass, matching the cascade’s inference budget\)\.

### 5\.1Main results

[Table˜3](https://arxiv.org/html/2607.07690#S5.T3)reports the main comparison on Qwen3\-0\.6B\. All deltas below are read from the*pass@1*column\. The trained pair methods and the refinement controls are two\-pass at inference; vanilla GRPO and zero\-shot are single\-pass \(see the footnote of[Table˜3](https://arxiv.org/html/2607.07690#S5.T3)\)\. Information exchange alone \(cooperative exchange:4646\) already improves over vanilla GRPO \(3030\) and over self\-refinement \(3232\), supporting the exchange claim; adding competition \(Agon:6161\) improves further over cooperation, supporting the competition claim\. The untrained MoA control is a two\-agent Mixture\-of\-Agents layer at inference\(Wang et al\.,[2024](https://arxiv.org/html/2607.07690#bib.bib13)\): each model reads the other’s attempt and refines its own\. It lifts GRPO by\+4\+4pp \(30→3430\\\!\\to\\\!34\); Agon lifts it by\+31\+31pp, roughly eight times the gain of untrained aggregation\. We note that the two levers contribute comparably: information exchange adds\+16\+16pp \(GRPO→\\tocooperative\) and competition a further\+15\+15pp \(cooperative→\\toAgon\)\. The dual\-adapter architecture alone contributes little: the competitive\-shared\-opponent variant sits at3232, barely above single\-model GRPO at3030, so the gain from exchange is attributable to the shared context, not to going from one model to two\.

Each method is a single training run; the confidence intervals in[Table˜4](https://arxiv.org/html/2607.07690#S5.T4)cover sampling variation over the 300 held\-out problems \(±∼5\.5\\pm\{\\sim\}5\.5pp\) but not run\-to\-run training variance\. Both headline deltas \(exchange:\+16\+16pp; competition:\+15\+15pp\) exceed the width of these intervals\. The Agon cascade reaches roughly twice vanilla GRPO’s held\-out pass@1 while*shortening*final\-stage traces; in gain terms, GRPO lifts the base by\+7\+7pp \(23→3023\\\!\\to\\\!30\) while Agon lifts it by\+31\+31pp over GRPO \(30→6130\\\!\\to\\\!61\), and\+26\+26pp over a two\-pass GRPO self\-cascade at matched inference budget, nearly doubling the untrained MoA cascade\.

To decompose the cascade gain, we evaluate the drafter adapter standalone after training \(plain prompt, no opponent summary\): it reaches 46 pass@1, a \+16 pp lift over vanilla GRPO\. This is notable given the budget asymmetry: each adapter drafts on only half the steps with group sizeNN, so it receives roughly a quarter of the standalone\-rollout gradient volume of the vanilla GRPO baseline \(which uses2​N2Non every problem\), plus challenger\-stream gradients on the alternating steps\. Our working hypothesis is that the challenger\-stream updates transfer to standalone solving \(the adapter is trained to verify and re\-derive, skills that a plain prompt also exercises\), but we have not tested this with a control \(e\.g\. the standalone accuracy of a cooperative\-trained or self\-refinement adapter\), so the attribution of the drafter’s \+16 pp to the competitive objective is a hypothesis, not an isolated measurement\. The challenger then converts a net 15 of the drafter’s 54 failures into correct answers \(with possible regressions among the drafter’s 46 successes\), yielding the final 61\. Under this reading, the drafter’s lift \(\+16 pp\) and the challenger’s conversion \(\+15 pp\) contribute comparable shares of the total gain, the former via the hypothesized transfer of challenger\-stream skills to standalone solving\.

To check that the gains are not an artifact of the training distribution,[Table˜2](https://arxiv.org/html/2607.07690#S5.T2)reports the same checkpoints on GSM8K and MATH\-500 without any further tuning: the ordering is preserved with smaller absolute margins than in\-distribution \(\+13\+13and\+19\+19pp over zero\-shot, vs\+38\+38on DeepMath\-hard\), as expected for a method whose pressure targets the hard regime\.

Table 2:Transfer sanity checks on Qwen3\-0\.6B \(pass@1, %; trained rows: single training run, same checkpoints and cascade protocol as[Table˜3](https://arxiv.org/html/2607.07690#S5.T3), no further tuning; zero\-shot: single evaluation; values rounded to nearest integer\)\.Table 3:Main comparison on Qwen3\-0\.6B, DeepMath\-hard held\-out \(%\), at a matched training generation budget; inference passes per method are noted in the footnote\. Single training run per method; 95% Clopper–Pearson confidence intervals over the 300\-problem held\-out set are shown in[Table˜4](https://arxiv.org/html/2607.07690#S5.T4)\. Values rounded to nearest integer\.pass@1 is the primary metric\. Pair\-trained methods and the refinement controls run two sequential generation passes per problem \(for Agon and cooperative: draft, then challenge reading the draft’s summary\); both cascade directions are evaluated and the better is reported\. Zero\-shot and vanilla GRPO are single\-pass\. avg\. len is the final\-stage completion length only \(full cascade uses∼\\sim2×\\timestokens at inference\)\. Zero\-shot and the untrained MoA involve no training and are reported as single evaluations\. 95% confidence intervals are Clopper–Pearson binomial over 300 held\-out problems\.

Table 4:95% Clopper–Pearson confidence intervals over the 300\-problem held\-out set \(single training run per method\)\.335577992020404060608080shorter traceszero\-shotGRPOMoAcoopAgonmean trace length \(k tokens, final stage\)held\-out pass@1 \(%\)\(a\) accuracy vs\. trace length05005001,0001\{,\}0001,5001\{,\}5002,0002\{,\}0002,5002\{,\}5003,0003\{,\}0002020404060608080training problems processed \(2​N2Nrollouts per problem for every method\)held\-out pass@1 \(%\)\(b\) over trainingGRPOself\-refinecoopAgonFigure 7:\(a\)Agon sits in the top\-left, higher accuracy at*shorter*traces \(lengths are final\-stage completions\), while GRPO buys length for little accuracy and the untrained MoA adds only a small bump\. The arrow is illustrative\.\(b\)Held\-out pass@1 at evaluation checkpoints over training\. Each point is a single sampled evaluation of a single training run \(binomial noise∼±5\.5\{\\sim\}\\pm 5\.5pp at 300 problems\); curves are not averaged over seeds\. Self\-refinement plateaus just above GRPO, cooperative exchange climbs higher, and competition\+\+exchange \(Agon\) separates clearly\. Agon is still improving at the end of the budget; we stop at the matched budget rather than at convergence\.
### 5\.2Scaling and generality

We expect smaller models, which live deeper in the hard regime, to benefit most\.[Table˜5](https://arxiv.org/html/2607.07690#S5.T5)shows the gain over vanilla GRPO shrinking but remaining positive as the base grows, and the same pattern holding on two other model families, Qwen3\.5\(Qwen Team,[2026](https://arxiv.org/html/2607.07690#bib.bib4)\)and Gemma 4\(Gemma Team,[2026](https://arxiv.org/html/2607.07690#bib.bib5)\): the delta tracks zero\-shot strength, not the family\. The Agon cascade on the 0\.6B base \(61\) exceeds not only zero\-shot Qwen3\-4B \(52\) but vanilla GRPO on the 4B base \(59\), a model∼7×\{\\sim\}7\\timeslarger, on the same held\-out set\. In every case the pair is two adapters over*one*base \(our instantiation, not a requirement of the method\); pairing across families is future work\.

Table 5:Model scaling and family generality: held\-out pass@1 \(%; trained cells: single training run, cascade protocol of[Table˜3](https://arxiv.org/html/2607.07690#S5.T3), rounded to the nearest point; zero\-shot: single evaluation\)\.
### 5\.3A second domain: competitive\-programming code

The recipe is not specific to math; any domain with a clean programmatic verifier qualifies\. We repeat the core comparison on Qwen3\-1\.7B trained on theeasy\-difficulty subset of CodeContests\(Li et al\.,[2022](https://arxiv.org/html/2607.07690#bib.bib6)\), withr​\(x,y\)=1r\(x,y\)=1iff the generated program passes all unit tests, a held\-out set of300300problems, and the same protocol and generation budget\.[Table˜6](https://arxiv.org/html/2607.07690#S5.T6)shows the same ordering as[Table˜3](https://arxiv.org/html/2607.07690#S5.T3), exchange beats the baseline and competition beats cooperation, with traces again shortening\. The absolute delta is smaller than on math: unit\-test rewards are sparser at this scale, and code traces make the opponent prefill longer relative to generation\. Across domains the claims should be read as directional rather than magnitude\-matched\.

Table 6:Second domain: CodeContests \(easydifficulty\) on Qwen3\-1\.7B, held\-out pass@1 \(%; single training run; zero\-shot: single evaluation; values rounded to nearest integer\)\.
### 5\.4Ablations

[Table˜7](https://arxiv.org/html/2607.07690#S5.T7)isolates each design choice on Qwen3\-0\.6B \(held\-out pass@1\)\. The competition claim is carried by the*competition*row \(adv vs coop\)\. The*information exchange*row \(Agon vs the shared\-opponent variant\) should be read with care: switching to a shared opponent removes the visible draft*and*, by the analysis in[Section˜4\.4](https://arxiv.org/html/2607.07690#S4.SS4), deactivates the competitive gradient, so this comparison changes two things at once and is a consistency check rather than an isolated exchange effect; the isolated exchange delta is GRPO vs cooperative in[Table˜3](https://arxiv.org/html/2607.07690#S5.T3)\. In the “shared opponent” ablation the challenger group is scored against one fixed opponent draft, hidden from its context \(group\-constantc​\(a\)c\(a\)\), so the conversion bonus normalizes away \([Section˜4\.4](https://arxiv.org/html/2607.07690#S4.SS4)\); the result \(3232vs3030\) is consistent with theory and shows that within\-group variance in opponent difficulty is required for the competitive term to contribute gradient\. A per\-rollout opponent pairing \(distinct hidden drafts\) is the honest ablation of competition without visible exchange; that variant is left for future work\. Role rotation is ablated explicitly: with fixed roles each adapter receives only one gradient stream, yielding 52 pass@1 versus 61 with rotation\.

Table 7:Ablations on Qwen3\-0\.6B \(held\-out pass@1, %; single training run per variant, rounded to the nearest point; best inbold\)\. Each variant changes a single*training\-time*choice\. The competitive\-shared\-opponent variant is evaluated single\-pass \(no draft to read\); all others use the two\-stage cascade protocol of[Table˜3](https://arxiv.org/html/2607.07690#S5.T3)\.
### 5\.5Analysis: why traces shorten

Empirically, Agon’s traces are shorter than GRPO’s \([Figure˜7](https://arxiv.org/html/2607.07690#S5.F7)a\): the final \(challenger\) stage averages3\.53\.5k tokens against8\.18\.1k for a single GRPO pass\. Note that this is a per\-stage comparison: the cascade also runs a drafting stage, whose length we do not report, so total generated tokens per problem are not directly compared here and a full token\-cost accounting is left for further analysis\. The like\-for\-like comparison is against the two\-stage untrained MoA control, where the same \(final\) stage is measured on both sides: Agon’s is half the length \(3\.53\.5k vs6\.96\.9k\)\. The headline reward contains no length term\. The observed shortening is consistent with the challenger having a candidate solution in context, so length\-heavy exploration is no longer the only route to a correct answer\. Challenger completions conditioned on a*correct*opponent summary are∼35%\{\\sim\}35\\%shorter than the challenger\-stage average \(2\.32\.3k vs3\.53\.5k tokens on the final checkpoint\), and[Figure˜6](https://arxiv.org/html/2607.07690#S4.F6)shows the typical pattern: the challenger pinpoints the flawed step, re\-derives it, and stops\.

### 5\.6An optional density lever \(auxiliary study\)

The shortening above is emergent: nothing in the headline reward asks for it\. The same game can, however, also grade brevity: instead of taxing tokens with a penalty, we make being correct in fewer tokens than the rival a way to win\. As a small side study, separate from the main recipe and from the two headline claims, we ask whether the shortening can thus be driven directly, using the length\-tiebreak term from[Section˜4\.4](https://arxiv.org/html/2607.07690#S4.SS4)\(a bonusγ\\gammawhen the challenger is correct*and*shorter than an equally\-correct opponent\)\. On Qwen3\-0\.6B, turning it on cuts mean trace length from3\.53\.5k to2\.62\.6k at essentially unchanged accuracy \(61→6061\\\!\\to\\\!60pass@1\), so the emergent shortening becomes directly controllable \([Table˜8](https://arxiv.org/html/2607.07690#S5.T8)\)\. Because the term only breaks ties among already\-correct traces, it compresses without measurably trading away accuracy\. We report this as an auxiliary knob; all headline numbers elsewhere use the length\-free reward\.

Table 8:Auxiliary density lever on Qwen3\-0\.6B \(pass@1, same cascade protocol\)\. The length tiebreak \([Section˜4\.4](https://arxiv.org/html/2607.07690#S4.SS4)\) shortens traces at near\-constant accuracy\. Not part of the headline recipe\.

## 6Conclusion

Agon trains two models to reason over each other’s work, and the pair co\-improves past where a single self\-graded policy stalls\. The competing peer supplies the signal outcome\-only RL lacks: it reads the policy’s solution and grades it implicitly, with no process labels and no change to the verifier\. Two distinct policies, trained head\-to\-head with rotating draft\-and\-challenge roles \(instantiated cheaply as two adapters over one frozen base\), let us test the idea\. Under a matched generation budget, information exchange beats vanilla GRPO and self\-refinement \(the exchange claim\), competition beats cooperation \(the competition claim\), and the resulting traces are shorter, a by\-product of the competition rather than of an explicit penalty\. At the same doubled inference budget, combining two copies of the base without training buys44points over GRPO, while training the pair to compete buys3131; the gain comes from the learned collaboration rather than from having two models\.

#### Limitations\.

Agon needs a verifier and reference problems, and we study math plus a smaller code study, both with clean programmatic rewards; noisier domains may behave differently\. Gains depend on the pair being matched in strength but divergent in behavior \(too large a gap collapses into distillation, too small a gap into self\-play\), and our divergence is maintained heuristically \(init\+\+rotation\) rather than guaranteed; complementarity is not quantified beyond the cascade win rate\. Text exchange bounds the channel bandwidth\. Inference is a two\-stage cascade, i\.e\. two sequential generation passes and added latency; reported trace lengths cover the final stage only, and the better cascade direction is selected post hoc on the held\-out set\. Compute parity holds for*generated*tokens only: the challenger’s extra prefill of the opponent summary is not equalized \([Section˜4\.5](https://arxiv.org/html/2607.07690#S4.SS5)\)\. Groups with zero within\-group variance \(all\-correct or all\-wrong\) yield undefinedσ\\sigmain GRPO advantage; such groups are dropped\. Every trained number in the paper is a single training run evaluated with a single sampled rollout per problem; the better cascade direction was selected post hoc on the held\-out set\. Run\-to\-run training variance is unquantified, and the reported deltas should be read with that in mind\. The reported magnitudes come from a hard, decontaminated subset and should not be read as DeepMath\-wide pass@1\.

#### Future work\.

This work establishes that the cross\-model signal helps while exchanging it in text; the channel itself is the next axis\. Exchanging hidden states directly, via KV\-cache injection or gated latent bridges between the two adapters, removes the serialization bottleneck and lets models share information they cannot verbalize\(Beliaev,[2025](https://arxiv.org/html/2607.07690#bib.bib29)\)\. Beyond accuracy, our auxiliary density lever \([Section˜5\.6](https://arxiv.org/html/2607.07690#S5.SS6)\) is a first step; scaling it into a full multi\-objective, compression traded explicitly against correctness, could make density a first\-class, tunable target, and a diversity term over divergent solution strategies could make the pair complementary enough to ensemble\.

## References

- Guo et al\. \[2025\]DeepSeek\-AI\. DeepSeek\-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\. 2025\.
- Shao et al\. \[2024\]Z\. Shao et al\. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models \(GRPO\)\. 2024\.
- Qwen Team \[2025\]A\. Yang et al\. Qwen3 Technical Report\. 2025\.
- Qwen Team \[2026\]Qwen Team\. Qwen3\.5: Towards Native Multimodal Agents\.[https://qwen\.ai/blog?id=qwen3\.5](https://qwen.ai/blog?id=qwen3.5), 2026\.
- Gemma Team \[2026\]Gemma Team, Google DeepMind\. Gemma 4 Technical Report\. 2026\.
- Li et al\. \[2022\]Y\. Li et al\. Competition\-Level Code Generation with AlphaCode\. Science, 2022\.
- He et al\. \[2025\]Z\. He et al\. DeepMath\-103K: A Large\-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset\. arXiv:2504\.11456, 2025\.
- Hu et al\. \[2022\]E\. Hu et al\. LoRA: Low\-Rank Adaptation of Large Language Models\. ICLR, 2022\.
- Aggarwal & Welleck \[2025\]P\. Aggarwal, S\. Welleck\. L1: Controlling How Long a Reasoning Model Thinks with Reinforcement Learning\. arXiv:2503\.04697, 2025\.
- Lightman et al\. \[2023\]H\. Lightman et al\. Let’s Verify Step by Step\. arXiv:2305\.20050, 2023\.
- Huang et al\. \[2024\]J\. Huang et al\. Large Language Models Cannot Self\-Correct Reasoning Yet\. ICLR, 2024\.
- Wang et al\. \[2023\]X\. Wang et al\. Self\-Consistency Improves Chain of Thought Reasoning in Language Models\. ICLR, 2023\.
- Wang et al\. \[2024\]J\. Wang et al\. Mixture\-of\-Agents Enhances Large Language Model Capabilities\. arXiv:2406\.04692, 2024\.
- Li et al\. \[2025\]W\. Li, Y\. Lin, M\. Xia, C\. Jin\. Rethinking Mixture\-of\-Agents: Is Mixing Different Large Language Models Beneficial? arXiv:2502\.00674, 2025\.
- Chen et al\. \[2024a\]X\. Chen et al\. Do NOT Think That Much for 2\+3=? On the Overthinking of o1\-Like LLMs\. arXiv:2412\.21187, 2024\.
- Yu et al\. \[2025\]Q\. Yu et al\. DAPO: An Open\-Source LLM Reinforcement Learning System at Scale\. arXiv:2503\.14476, 2025\.
- Chen et al\. \[2024b\]Z\. Chen et al\. Self\-Play Fine\-Tuning Converts Weak Language Models to Strong Language Models \(SPIN\)\. ICML, 2024\.
- Yuan et al\. \[2024\]W\. Yuan et al\. Self\-Rewarding Language Models\. ICML, 2024\.
- Zhao et al\. \[2025\]A\. Zhao et al\. Absolute Zero: Reinforced Self\-Play Reasoning with Zero Data\. arXiv:2505\.03335, 2025\.
- Huang et al\. \[2025\]C\. Huang et al\. R\-Zero: Self\-Evolving Reasoning LLM from Zero Data\. arXiv:2508\.05004, 2025\.
- Du et al\. \[2023\]Y\. Du et al\. Improving Factuality and Reasoning in Language Models through Multiagent Debate\. arXiv:2305\.14325, 2023\.
- Madaan et al\. \[2023\]A\. Madaan et al\. Self\-Refine: Iterative Refinement with Self\-Feedback\. NeurIPS, 2023\.
- Cheng et al\. \[2024\]P\. Cheng et al\. Self\-Playing Adversarial Language Game Enhances LLM Reasoning \(SPAG\)\. NeurIPS, 2024\.
- Goodfellow et al\. \[2014\]I\. Goodfellow et al\. Generative Adversarial Nets\. NIPS, 2014\.
- Irving et al\. \[2018\]G\. Irving, P\. Christiano, D\. Amodei\. AI Safety via Debate\. arXiv:1805\.00899, 2018\.
- Kirchner et al\. \[2024\]J\. H\. Kirchner et al\. Prover\-Verifier Games Improve Legibility of LLM Outputs\. arXiv:2407\.13692, 2024\.
- Subramaniam et al\. \[2025\]V\. Subramaniam et al\. Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains\. ICLR, 2025\.
- Stanford HAI \[2026\]Stanford HAI\. The 2026 AI Index Report, Chapter 2: Technical Performance\. Stanford University, 2026\.
- Beliaev \[2025\]V\. Beliaev\. Esperantix: Inference\-Time Composition of Heterogeneous Language Models via Latent\-Space Communication\. Working draft, 2025\.
- von Werra et al\. \[2020\]L\. von Werra et al\. TRL: Transformer Reinforcement Learning\.[https://github\.com/huggingface/trl](https://github.com/huggingface/trl), 2020\.
- Kwon et al\. \[2023\]W\. Kwon et al\. Efficient Memory Management for Large Language Model Serving with PagedAttention \(vLLM\)\. SOSP, 2023\.
- Hendrycks et al\. \[2021\]D\. Hendrycks et al\. Measuring Mathematical Problem Solving with the MATH Dataset\. NeurIPS, 2021\.
- Cobbe et al\. \[2021\]K\. Cobbe et al\. Training Verifiers to Solve Math Word Problems \(GSM8K\)\. arXiv:2110\.14168, 2021\.