Benchmarks Don't Talk Back: Learning Through Self-Disagreement
Summary
The paper introduces 'dialectical competence' — knowing when to hold or yield a position under counter-reasoning — and proposes Self-Disagreement, an RLVR method where a model generates its own antithesis and is trained on ruling over it, with no second model, reward model, or human annotation.
View Cached Full Text
Cached at: 11/16/26, 08:19 AM
# Benchmarks Don’t Talk Back:Learning Through Self-Disagreement
Source: [https://arxiv.org/html/2611.17563](https://arxiv.org/html/2611.17563)
###### Abstract
A benchmark usually measures a language model alone with a question, and deployment rarely looks like that: answers get contested, and models give up correct ones under counter\-reasoning that is itself wrong\. We show that what fails there is not a shortfall of accuracy but a capability in its own right — knowing when to hold a position and when to yield to a better one — which we call*dialectical competence*\. It stands on first\-answer accuracy without following from it: a model can be made materially better at answering and no better at this, and training for accuracy alone can drift against it\. Nor can it be bought at inference, by self\-reflection or by sampling more\. Taking a cue from the dialectical tradition in philosophy, in which productive opposition arises within a position rather than being imposed on it from outside, we hypothesize that this capability can be trained from opposition the model supplies itself\. It states a position, generates the strongest case against that position — its*antithesis*— rules on whether the case stands, and is trained on that ruling, with no second model at training or at deployment\. We call this learning through*Self\-Disagreement*: reinforcement learning from verifiable rewards throughout, where the same check that scores an answer against its reference scores the ruling too, so no reward model, no preference data and no human annotation enter anywhere\. The capability is trained rather than elicited: weak in the base model, it arrives with the reward, is not paid for out of first\-answer accuracy, and holds against challengers the model never trained against, including ones stronger than itself\. The opposition a model needs in order to improve, it turns out, need not come from outside it\.
## 1Introduction
Benchmarks do not talk back:Language models have become markedly better at solving problems across successive open frontier families\([Grattafiori et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib11);[Yang et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib48)\), with reinforcement learning against answers a program can check providing one route to that improvement\([DeepSeek\-AI et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib5);[Shao et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib36);[Yu et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib49);[Liu et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib27)\)\. Benchmark suites register that progress across general knowledge\([Hendrycks et al\., 2020](https://arxiv.org/html/2611.17563#bib.bib15)\), graduate\-level science\([Rein et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib34)\), competition mathematics\([Hendrycks et al\., 2021](https://arxiv.org/html/2611.17563#bib.bib16);[Gao et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib7)\)and code\([Jain et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib18)\)\. Yet each ultimately reduces evaluation to a question, a model’s answer, and a score on that answer\. Deployment is often different: users push back on assistants; inside agentic loops another process may propose a different answer and contest a draft before it is accepted\([Qian et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib33);[Hong et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib17);[Wu et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib47)\); and later turns may ask a model to reconsider an answer it has already given\. Disagreement of that kind is not hostile and not an edge case; it is the normal texture of use\. Answers, however, move under it: models accommodate a user’s stated view rather than the evidence\([Sharma et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib37)\), give up correct answers when merely doubted\([Laban et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib22)\), and abandon answers that were right under sustained counter\-reasoning\([Wang et al\., 2023a](https://arxiv.org/html/2611.17563#bib.bib42)\), at rates that do not fall as the model gets stronger\([Abbasloo, 2025](https://arxiv.org/html/2611.17563#bib.bib1)\)\.
Figure 1:Answering better does not buy the capability\.Share of the base model’s correct answers each arm gives up under challenge, one position per challenger; each trained arm is the mean of three training seeds\. Each of the three ordinarily trained arms answers better on its own and still tracks the base model here, as does SCoRe \([Kumar et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib21); App\.[N](https://arxiv.org/html/2611.17563#A14)\); only our design, Self\-Disagreement \(SelfD\-RL\), separates from them — by 25\.3 pp on Qwen3\-8B and 36\.1 pp on Gemma\-2 9B on average across these eleven frontier challengers \(up to 45 and 67 pp from the base model; more in Table[8](https://arxiv.org/html/2611.17563#A9.T8)\)\.A capability in its own right:What moves under challenge is not another shortfall in answering, because the failure begins after a correct answer has already been produced\. What is missing is the ability to weigh a case made against a position and to adjust only when the case deserves it — holding when the challenge is incorrect, yielding when it is correct — which we call*dialectical competence*\. Training for correctness does not supply it, though correctness is its precondition: a model that cannot reach the right answer has nothing to hold\. It is the higher skill, standing on answering rather than competing with it\. It is also the capability hard to supply from outside, because every way of correcting a model that leans on a stronger judge — a larger model, a separate verifier, a person — needs that judge to exist, and one better than the model being corrected is rare and costly\. To separate the two, we train on two base models drawn from different families, Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib48)\)and Gemma\-2 9B\([Gemma Team et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib9)\), and run the same comparison on each\. Figure[1](https://arxiv.org/html/2611.17563#S1.F1)places the base model beside three ordinarily trained arms \(each defined in §[B](https://arxiv.org/html/2611.17563#A2)\), against the same challengers: each answers better on its own, and each still gives up its correct answers under challenge at close to the base model’s rate\. Training against the model’s own opposition separates from that pattern, on both families \(the diagnosis is made precise in §[2](https://arxiv.org/html/2611.17563#S2)\)\.
Our proposal — learning through Self\-Disagreement:The remedies that suggest themselves share one defect: every one of them works on the answer, when the capability at issue is the further one that begins after the answer\. Training harder for correctness, giving the model a second pass over its own answer, and training it to judge that answer before committing all track the base model rather than separating from it \(Figure[1](https://arxiv.org/html/2611.17563#S1.F1)\); buying more inference — a reconsideration pass, or sampling many answers and voting — does not reach it either \(§[4](https://arxiv.org/html/2611.17563#S4)\); Appendix[A](https://arxiv.org/html/2611.17563#A1)discusses these lines of work at greater length\. In each, the move the model makes is prompted, sampled or borrowed; it never follows from the model’s own ruling on a case it was made to confront\. The dialectical tradition puts exactly that ruling at the center: a position is developed by meeting the strongest case against it\([Plato, 1963](https://arxiv.org/html/2611.17563#bib.bib31);[Popper, 2005](https://arxiv.org/html/2611.17563#bib.bib32)\); that case is raised from within the position rather than handed to it from above\([Hegel, 2018](https://arxiv.org/html/2611.17563#bib.bib14)\); and the position must rule on the case before it may move\. We take this as a training paradigm rather than as a metaphor\. A single policy plays both parts, differing only in what each is shown\. It states a position, the*thesis*; in a context of its own, holding the question and the case just made but never the reference answer, it argues against that position, the*antithesis*; it then returns to the answering role, rules on whether that case stands, and commits to a final answer, the*synthesis*\(§[3](https://arxiv.org/html/2611.17563#S3)\)\. Only the answering turns are rewarded, and the reward is the verifier we already have: the same check that scores an answer against its reference also scores the answer the model settles on once opposed\. No reward model, no preference data, no second model at training or deployment:*the opposition a model needs, it supplies itself*\. We contribute:
1. \(1\)*Dialectical competence as a capability in its own right*: we measure it on a probe built for it, and show that neither accuracy\-centered training nor inference\-time remedies supply it \(§[2](https://arxiv.org/html/2611.17563#S2)\)\.
2. \(2\)*Self\-Disagreement*: a single\-policy extension of RLVR, with the opposition drawn from the policy being trained \(§[3](https://arxiv.org/html/2611.17563#S3)\)\.
3. \(3\)*Demonstrating that Self\-Disagreement improves it*: trained on two base families, the models gain dialectical competence without paying for it in first\-answer accuracy \(§[4\.1](https://arxiv.org/html/2611.17563#S4.SS1)\), and keep the gain against challengers they never trained against, with an account of where it comes from \(§[4\.2](https://arxiv.org/html/2611.17563#S4.SS2)\)\.
## 2The problem
A probe that makes holding and yielding visible:Benchmarks record whether a model ended up right, not what it did when it was pushed, so we measure on a probe built for the second question\. We call it the*disagreement probe*, or*D\-probe*, and it is used for every measurement of behavior under challenge in this paper\. It is 327 questions, each put to every one of 17 challengers — 15 external models spanning open checkpoints to frontier systems, the model’s own frozen starting point, and its own critique\. The diagnosis below runs it on the two base models, Qwen3\-8B and Gemma\-2 9B\. Every model answers once and that answer is replayed to each challenger, so only the answer given after the challenge can differ, and all three answers are graded: the model’s own, the challenger’s, and the one the model settles on\. Knowing which of the first two was right is what makes that choice legible, because it says what the right action was\. What we report is the accuracy of that settled answer, the*synthesis accuracy*\.
The label is checked, not assumed:Synthesis accuracy is what the metric rests on, so it was checked a second way: on the items where a base model gives up a correct answer, a blinded jury of Claude\-Opus\-4\.7, GPT\-5\.5 and Gemini\-3\.1\-pro\-preview reads the reasoning and judges all 121 such items to be non\-genuine — conceding and adopting the challenger’s answer with no valid pushback, or merely restating it — rather than a differently argued answer\. What synthesis accuracy loses is therefore reasoning under challenge, not a relabelled answer \(§[G](https://arxiv.org/html/2611.17563#A7)\)\.
Both base models are one\-sided:What their synthesis accuracy is made of, rather than its level, is the diagnosis\. On the items where the model was right and the challenger wrong — where holding is the correct action — Qwen3\-8B ends on the correct answer 53\.3 % of the time and Gemma\-2 9B 25\.1 %\. On the items where the challenger was right and the model wrong, where yielding is correct, the same two models score 84\.1 % and 93\.2 %\. That pairing is not discrimination between good arguments and bad; it is close to unconditional surrender, and one behavior produces both numbers\. The cost rises with the challenger: across Figure[1](https://arxiv.org/html/2611.17563#S1.F1)’s challengers the base model gives up 17 % to 61 % of its correct answers on Qwen3\-8B and 25 % to 83 % on Gemma\-2 9B\.
Figure 2:Buying accuracy is not buying the capability\.SuperGPQA: each arm sits at its thesis accuracy \(xx\) against how often it gave up a correct answer under challenge \(yy, inverted\), so better is up and to the right\. Bars are±1\\pm 1sd across three training seeds \(for Base, over its eight sampled answers\)\.Training for accuracy does not reach it:The natural reading of that is that the models are not yet good enough, and that better answering will carry the rest\. It does not\. A control trained for accuracy alone \(Accuracy\-RL; defined, with its settings, in §[B](https://arxiv.org/html/2611.17563#A2)\), from the same base model and the same data, also improves the first answer and buys none of the behavior under challenge, which can drift against it over training rather than holding steady \(Appendix[E](https://arxiv.org/html/2611.17563#A5)\)\. Figure[2](https://arxiv.org/html/2611.17563#S2.F2)shows it on SuperGPQA\([M\-A\-P Team et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib28)\): the trained alternatives, the published self\-correction method SCoRe\([Kumar et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib21)\)among them, move right, to higher thesis accuracy, and stay at the same height\. Answering well is the floor, not the capability: a model that cannot produce the right answer has none to hold, but past that floor more of it buys nothing here\. Nor is the opposition’s content incidental: an opposition that carries no content recovers only part of the gain \(§[4\.2](https://arxiv.org/html/2611.17563#S4.SS2)\)\.
What that leaves:The failure is not a shortfall that better answering removes, and it is not repaired by the inference\-time remedies we test, self\-consistency\([Wang et al\., 2022](https://arxiv.org/html/2611.17563#bib.bib44)\)and self\-reflection\([Madaan et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib30)\), or by training the same objective harder\. What is missing is the decision itself: telling a correct challenge from an incorrect one and acting on the difference\. The rest of this paper is about training that\.
## 3Design
Overview:We train a single policy under a dialectical protocol we call*Self\-Disagreement*\. The policy commits to an answer, argues against that answer in a separate context, then returns to rule on the argument before committing again\. Only the answering turns are trained\. The protocol extends RLVR rather than departing from it: the verifier that scores an answer against its reference also scores the ruling, and nothing else in the training loop changes\.
Where the opposition comes from:Learning to adjudicate disagreement presupposes a source of disagreement\. The usual construction takes it from a second model, which makes the training signal a property of the*pair*: an opposition that is too weak can be dismissed without discrimination, and one that is too strong makes deference the best available response\. The policy’s competence also changes as it trains, so a pairing calibrated at the start does not stay calibrated, and keeping it so means tracking a quantity that is not directly observable\. Drawing the opposition from the policy itself removes that coupling\. Its competence tracks the policy’s by construction and without scheduling, every training item has an opposition available without a second model to serve it, and what is deployed is a single set of weights\. The construction does rest on an assumption worth stating: that a policy can be induced to argue against a position it has just committed to\.
1:policy
πθ\\pi\_\{\\theta\}, RLVR verifier
𝒱\(⋅,y⋆\)\\mathcal\{V\}\(\\cdot\\,;y^\{\\star\}\), group size
GG, admission band
\[p¯,p¯\]\[\\underline\{p\},\\overline\{p\}\]
2:
3:sample a batch of questions
qqwith
p0\(q\)∈\[p¯,p¯\]p\_\{0\}\(q\)\\in\[\\underline\{p\},\\overline\{p\}\]
4:for all
qqin the batchdo
5:for
i=1i=1to
GGdo⊳\\trianglerightone GRPO group
6:
T∼πθ\(⋅∣q\)T\\sim\\pi\_\{\\theta\}\(\\cdot\\mid q\)⊳\\trianglerightanswering role
7:
A∼πθ\(⋅∣q,T\)A\\sim\\pi\_\{\\theta\}\(\\cdot\\mid q,T\)⊳\\trianglerightopposing role; noy⋆y^\{\\star\}
8:
S∼πθ\(⋅∣q,T,A\)S\\sim\\pi\_\{\\theta\}\(\\cdot\\mid q,T,A\)⊳\\trianglerightresumed: rule, then answer
9:
cT←𝒱\(T,y⋆\)c\_\{T\}\\leftarrow\\mathcal\{V\}\(T;y^\{\\star\}\);
cS←𝒱\(S,y⋆\)c\_\{S\}\\leftarrow\\mathcal\{V\}\(S;y^\{\\star\}\)⊳\\trianglerightthe answers
10:
cv←𝟏\[v=𝒱\(A;y⋆\)\]c\_\{v\}\\leftarrow\\mathbf\{1\}\[v=\\mathcal\{V\}\(A;y^\{\\star\}\)\]⊳\\trianglerightand the ruling
11:
β←2\.0\\beta\\leftarrow 2\.0if
cT=0∧𝒱\(A,y⋆\)=0c\_\{T\}=0\\wedge\\mathcal\{V\}\(A;y^\{\\star\}\)=0, else
11
12:
R\(τi\)←βcS\+wvcv\+wTcTR\(\\tau\_\{i\}\)\\leftarrow\\beta c\_\{S\}\+w\_\{v\}c\_\{v\}\+w\_\{T\}c\_\{T\}
13:
\-λ\(cap\+stub\)−wϕ\(ϕT\+ϕv\)\\phantom\{R\(\\tau\_\{i\}\)\}\\mathrel\{\-\}\\lambda\(\\mathrm\{cap\}\+\\mathrm\{stub\}\)\-w\_\{\\phi\}\(\\phi\_\{T\}\+\\phi\_\{v\}\)
14:if
ϕT∨ϕv\\phi\_\{T\}\\vee\\phi\_\{v\}then
R\(τi\)←RfloorR\(\\tau\_\{i\}\)\\leftarrow R\_\{\\mathrm\{floor\}\}⊳\\trianglerightoverride
15:endif
16:endfor
17:
A^i←\(R\(τi\)−μq\)/σq\\hat\{A\}\_\{i\}\\leftarrow\\big\(R\(\\tau\_\{i\}\)\-\\mu\_\{q\}\\big\)/\\sigma\_\{q\}⊳\\trianglerightgroup\-relative
18:endfor
19:update
θ\\thetaon the clipped surrogate, KL\-penalized in the return⊳\\trianglerightstock GRPO
Algorithm 1Self\-Disagreement, one training step
one policyπθ\\pi\_\{\\theta\}, two rolesanswering role • rewardedq→q\\;\\rightarrow\\;thesisTTopposing role • inference\-onlyantithesisAAown context, noy⋆y^\{\\star\}, not a targetanswering role resumed • rewardedverdict→\\;\\rightarrow\\;synthesisSSq,Tq,Tq,T,Aq,T,AreturnR\(τ\)R\(\\tau\)holding, yielding, correctionupdate
Figure 3:Self\-Disagreement\.Two roles of one policy, in separate contexts\. The opposing turn is sampled from the same weights tracking the policy as it trains\. It conditions the synthesis, but is never itself a training target, so it reaches the answering role as given, the way a user’s disagreement would\. The answering role resumes and must rule before it may commit\. Settings in §[4](https://arxiv.org/html/2611.17563#S4)\.
The dialectical exchange:Figure[3](https://arxiv.org/html/2611.17563#S3.F3)shows one exchange and Algorithm[1](https://arxiv.org/html/2611.17563#alg1)one training step; the prompt templates for all three turns are printed verbatim in §[C](https://arxiv.org/html/2611.17563#A3)\. The policy is prompted into two roles that share its weights and differ only in what each is shown\. In the answering role it writes the*thesis*, and later the*synthesis*; in the opposing role it writes the*antithesis*, disputing the strategy, the decisions and the reasoning of that particular answer*even where they appear correct*rather than making an independent second attempt at the question\. Opposition is therefore unconditional — a correct thesis is argued against as readily as an incorrect one — which is what makes*holding*a position something the policy must learn, since a policy opposed only when it was wrong would never need to tell a sound challenge from an unsound one\. Letπθ\\pi\_\{\\theta\}be the policy and letq∼𝒟q\\sim\\mathcal\{D\}be a question with reference answery⋆y^\{\\star\}\. One dialectical exchange is a trajectoryτ=\(T,A,S\)\\tau=\(T,A,S\)of three generations from the*same*parameters under two role prompts — answering, then opposing, then answering again under instruction to judge:
πθ\(τ∣q\)=πθ\(T∣q\)⏟thesisπθ\(A∣q,T\)⏟antithesis \(sampled, inference\-only\)πθ\(S∣q,T,A\)⏟synthesis,\\pi\_\{\\theta\}\(\\tau\\mid q\)\\;=\\;\\underbrace\{\\pi\_\{\\theta\}\(T\\mid q\)\}\_\{\\text\{thesis\}\}\\;\\underbrace\{\\pi\_\{\\theta\}\(A\\mid q,T\)\}\_\{\\text\{antithesis \(sampled, inference\-only\)\}\}\\;\\underbrace\{\\pi\_\{\\theta\}\(S\\mid q,T,A\)\}\_\{\\text\{synthesis\}\},\(1\)The synthesisSSis internally ordered: it opens with a verdictv∈\{0,1\}v\\in\\\{0,1\\\}onAA’s answer and only then commits to an answer\. Because the answer is decoded after the verdict and conditioned on it, the policy must adjudicate before it may answer, andvvis separately verifiable; ablating the verdict removes that opening block, leaving a synthesis that answers with no obligation to rule\. Writing𝒱\(⋅,y⋆\)∈\{0,1\}\\mathcal\{V\}\(\\cdot\\,;y^\{\\star\}\)\\in\\\{0,1\\\}for the ordinary RLVR verifier — exact match against the reference answer, so that no learned reward model and no preference data enter anywhere — the trajectory inducescT=𝒱\(T,y⋆\)c\_\{T\}=\\mathcal\{V\}\(T;y^\{\\star\}\),cS=𝒱\(S,y⋆\)c\_\{S\}=\\mathcal\{V\}\(S;y^\{\\star\}\), and
cv=\[v=𝒱\(A;y⋆\)\],c\_\{v\}\\;=\\;\\mathbf\{1\}\\\!\\left\[\\,v=\\mathcal\{V\}\(A;y^\{\\star\}\)\\,\\right\],\(2\)which scores whether the policy judged the opposition rightly, not whether the opposition was right\. This is where the verifier’s reach is extended: the same check, applied to a decision about the opposition’s answer rather than to an answer\. The credit is available on every item, including those in which the opposition concedes, so the objective by itself does not rule out a policy that agrees whenever the opposition agrees with it\. It is more than that: removing the content of the disagreement, together with the supervision that depends on that content, costs roughly half to two thirds of the gain, and the trained policy keeps its advantage against challengers it never trained against \(§[F](https://arxiv.org/html/2611.17563#A6)\)\. The antithesis is sampled from the policy but is*inference\-only*: its trajectory is never collected into the training episode, soπθ\(A∣q,T\)\\pi\_\{\\theta\}\(A\\mid q,T\)is a sampling distribution rather than an optimized factor, and opposing is never rewarded — only answering and adjudicating are — which removes any route by which it could learn to soften its challenges\. No factor of Eq\. \([1](https://arxiv.org/html/2611.17563#S3.E1)\) is conditioned ony⋆y^\{\\star\}: the verifier is applied only after the trajectory is complete, so it cannot oppose selectively on items it is about to answer incorrectly\. Thesis and synthesis are optimized as one trajectory, not as separate turns, so credit assignment reaches the answer and the ruling on its challenge jointly\.
Guarding against collapse:The characteristic failure of an endogenous construction is trivial agreement, in whichAAconcedesTTand the dialectic degenerates to a single pass\. Two provisions address this, neither of them part of the objective\. The first is uninformedness, above: the opposing role cannot tell which items it would be safe to concede\. The second is the admission band — items enter training only when the base policy’s pass ratep0\(q\)=Pr\[𝒱\(T;y⋆\)=1∣q;πθ0\]p\_\{0\}\(q\)=\\Pr\[\\mathcal\{V\}\(T;y^\{\\star\}\)=1\\mid q;\\pi\_\{\\theta\_\{0\}\}\]lies in\[p¯,p¯\]\[\\underline\{p\},\\overline\{p\}\], which excludes items on which opposition is respectively vacuous or futile\. What the opposition and the verdict actually do in training is then an empirical question rather than a guaranteed one, and we treat it as such: Appendix[F](https://arxiv.org/html/2611.17563#A6)measures the opposition the policy actually produces, and §[4](https://arxiv.org/html/2611.17563#S4)measures the resulting behavior rather than relying on the construction\.
At inference:None of the training machinery is required to use the result\. The trained policy is an ordinary model: it answers a question in one pass, and what training has changed is how it behaves*when it is disagreed with*\. The evaluation of §[4](https://arxiv.org/html/2611.17563#S4)therefore presents the trained policy with an opposing view and measures the answer it commits to, the same situation the protocol trains, with none of the environment’s apparatus\. There is no admission band, no candidate selection, no verifier and no reference answer, and, since the opposing role is the policy itself, no second model to deploy\.
The objective:Two failures are penalized directly:
𝟏\[cT=1,cS=0\]⏟cap\(capitulation\)𝟏\[cT=0,𝒱\(A,y⋆\)=1,cS=0\]⏟stub\(stubbornness\)\\underbrace\{\\mathbf\{1\}\[\\,c\_\{T\}=1,\\;c\_\{S\}=0\\,\]\}\_\{\\mathrm\{cap\}\\;\\text\{\(capitulation\)\}\}\\qquad\\underbrace\{\\mathbf\{1\}\[\\,c\_\{T\}=0,\\;\\mathcal\{V\}\(A;y^\{\\star\}\)=1,\\;c\_\{S\}=0\\,\]\}\_\{\\mathrm\{stub\}\\;\\text\{\(stubbornness\)\}\}\(3\)Capitulation is a correct thesis surrendered; stubbornness is a valid challenge refused\.*Synthesis accuracy*,Pr\(cS=1\)\\Pr\(c\_\{S\}\{=\}1\), is the outcome variable of every result in this paper\. The return, for the configuration reported throughout, is
R\(τ\)=βcS\+wvcv\+wTcT−λcapcap−λstubstub−wϕ\(ϕT\+ϕv\),R\(\\tau\)\\;=\\;\\beta\\,c\_\{S\}\\;\+\\;w\_\{v\}c\_\{v\}\\;\+\\;w\_\{T\}c\_\{T\}\\;\-\\;\\lambda\_\{\\mathrm\{cap\}\}\\,\\mathrm\{cap\}\\;\-\\;\\lambda\_\{\\mathrm\{stub\}\}\\,\\mathrm\{stub\}\\;\-\\;w\_\{\\phi\}\\big\(\\phi\_\{T\}\+\\phi\_\{v\}\\big\),\(4\)subject to a hard format gate: if either the thesis or the verdict fails to parse \(ϕT∨ϕv\\phi\_\{T\}\\vee\\phi\_\{v\}\), the return is*replaced*by a floor value and every positive channel is discarded,
R\(τ\)←RfloorifϕT∨ϕv\.R\(\\tau\)\\;\\leftarrow\\;R\_\{\\mathrm\{floor\}\}\\quad\\text\{if \}\\phi\_\{T\}\\vee\\phi\_\{v\}\.\(5\)Hereλcap=λstub=0\.5\\lambda\_\{\\mathrm\{cap\}\}=\\lambda\_\{\\mathrm\{stub\}\}=0\.5,wT=1w\_\{T\}=1,wv=0\.4w\_\{v\}=0\.4,wϕ=0\.1w\_\{\\phi\}=0\.1andRfloor=−0\.75R\_\{\\mathrm\{floor\}\}=\-0\.75\. The objective keeps paying for a correct first answer at full weight while adding the ruling at0\.40\.4: the capability is trained on top of answering, not in place of it\. The gate is worth more than its weight suggests: the policy cannot decline to adjudicate\. The remaining coefficient,β\\beta, multiplies the synthesis credit rather than adding to it:β=2\.0\\beta=2\.0whencT=0c\_\{T\}=0and𝒱\(A,y⋆\)=0\\mathcal\{V\}\(A;y^\{\\star\}\)=0, and11otherwise, since that is the one cell where no correct position exists to defer to and the answer has to be reached rather than chosen\. The symmetry ofλcap\\lambda\_\{\\mathrm\{cap\}\}andλstub\\lambda\_\{\\mathrm\{stub\}\}is load\-bearing\. An objective on terminal correctness alone is invariant to the trajectory by which correctness is reached, including trajectories that discard a correct answer and recover it — which §[4](https://arxiv.org/html/2611.17563#S4)shows accuracy\-only training does not merely tolerate but increasingly takes\. Penalizing capitulation alone would buy stability at the cost of revisability\. Optimization is by GRPO\([Shao et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib36)\): for eachqqa group ofGGtrajectories is sampled and scored by its group\-relative advantageA^i=\(R\(τi\)−μq\)/σq\\hat\{A\}\_\{i\}=\(R\(\\tau\_\{i\}\)\-\\mu\_\{q\}\)/\\sigma\_\{q\}, maximized under a clipped surrogate\([Schulman et al\., 2017](https://arxiv.org/html/2611.17563#bib.bib35)\), with the KL penalty to the reference policy applied to the return rather than to the loss\. The algorithm is unmodified: the dialectic changes what is scored, not how the update is computed\.
## 4Evaluation
This section has three parts\. The first asks whether the capability is there at all, and whether available approaches produce it: it is weak in the base model, arrives with training, holds on benchmarks the training never saw, and no rival design or test\-time scheme we compare reaches it\. The second asks where the gain comes from: whether the model answers better on the first attempt or decides better once challenged, and which part of the design buys it\. The third asks whether the result is valid and generalizes: a second reading of the metric by a blinded jury of three frontier models, a different base model, and challengers it never trained against\.
Setup:Every trained arm is reinforcement learning from verifiable rewards on the same data, from the same initialization and with the same verifier; what separates them is the return and the number of turns it is computed over\. Two further arms are not trained at all: the test\-time schemes, which spend additional inference on the base model instead\. Every arm, the base model included, meets the challenge under the same instruction, which asks for a verdict on it before the final answer; without that request the other arms do no better, and on Qwen3\-8B worse \(App\.[C](https://arxiv.org/html/2611.17563#A3)\)\. Because the opposition is sampled rather than fixed we resample it over independent draws \(eight; four in Table[3](https://arxiv.org/html/2611.17563#S4.T3)\); in the main comparisons each trained arm is the mean of three training seeds and its spread is across seeds, the base model’s across its draws\. Capitulation and stubbornness, defined in §[3](https://arxiv.org/html/2611.17563#S3), are counted over every scored item rather than over the subset on which each could occur, so that they add to synthesis accuracy and to each other\. Behavior under challenge is measured on the D\-probe throughout; the benchmark panel asks whether what it measures survives away from it\. What each arm is, the model, data, hardware and optimizer settings, the step at which each arm is read, and the construction of the disagreement probe are in §[B](https://arxiv.org/html/2611.17563#A2)\.
### 4\.1Overall results and analysis
Does it work, and could anything already available have done it:The capability is not present in either base model and it arrives with training in both \(Figure[4](https://arxiv.org/html/2611.17563#S4.F4)\)\. The single\-pass accuracy\-trained arm \(Accuracy\-RL\) is flat over its own run, so more training compute does not close the gap; nor does spending more at inference \(the test\-time columns of Table[1](https://arxiv.org/html/2611.17563#S4.T1)\)\. Table[1](https://arxiv.org/html/2611.17563#S4.T1)then asks whether what was learned on the probe survives away from it\.
Figure 4:Dialectical competence is trained under Self\-Disagreement, while the arms that drop the disagreement recover at most part of it\.Bands across seeds\. Separate models: read the shapes, not the cells\.It survives off the training probe, on both base models:On Qwen3\-8B the trained model is ahead of the base model on all nine public benchmarks, on mathematics and code as well as on the multiple\-choice sets where a challenge is cheapest to issue, and on average it settles on the right answer after being challenged \+5\.90 pp more often \(95 % CI over the nine benchmarks\[\+4\.64,\+7\.16\]\[\+4\.64,\+7\.16\]; over the three training seeds\[\+5\.00,\+6\.80\]\[\+5\.00,\+6\.80\]\), 50\.0 % to 55\.9 %\. The final row of the upper block of Table[1](https://arxiv.org/html/2611.17563#S4.T1)repeats the whole exercise on a second base model, Gemma\-2 9B, and the pattern holds: 19\.9 % to 37\.8 % on average, ahead of its own base on every benchmark \(§[L](https://arxiv.org/html/2611.17563#A12)\)\. Every column of the upper block is read under the same protocol and scored by the same verifier, so what separates the columns is what each was trained to do and not how each was measured\.
How it works with external challengers:The upper block of Table[1](https://arxiv.org/html/2611.17563#S4.T1)lets each arm meet its own antithesis, which is how the model is used; the lower block holds the challenge fixed, giving every arm the same first answer and the same antithesis from Gemini\-3\.8\-flash or Claude\-Haiku\-4\.5, so that only the decision is the arm’s own \(App\.[Q](https://arxiv.org/html/2611.17563#A17)\)\. The lead survives, 6\.2 to 14\.1 points over the base model with every 95 % interval over training seeds above 5, while no other arm, trained on another objective or given more inference, gains more than 1\.4 points on it, and the reason lies in how each responds to the same challenge: the other arms take most antitheses put to them, right or wrong, much as the base model does, whereas Self\-Disagreement holds a right first answer against a wrong antithesis more than twice as often as any of them and still takes most of the right ones\. Nor does it win by switching less: where the base model, its final answer biased toward its first, switches as rarely as Self\-Disagreement, it still trails it by 7\.5 and 4\.4 points of synthesis accuracy onGemma\-2 9B, one per outside challenger \(App\.[Q](https://arxiv.org/html/2611.17563#A17)\), so what training on the challenge adds is which items the model holds on, not how many, which is what telling a correct challenge from an incorrect one looks like once the challenge itself is the same for everyone\.
Table 1:Synthesis accuracy under challenge, per benchmark \(%\)\.*Vf\-RL*,*Rv\-RL*and*Ac\-RL*are Verify\-RL, Revise\-RL and Accuracy\-RL\.*SCoRe*is the method of[Kumar et al\. \(2024\)](https://arxiv.org/html/2611.17563#bib.bib21), detailed in App\.[N](https://arxiv.org/html/2611.17563#A14)\.*Maj@64*and*Self\-Refl*: the base model given more inference — test\-time self\-consistency and test\-time self\-reflection \(details in App\.[J](https://arxiv.org/html/2611.17563#A10)\)\.‡Per\-benchmark detail in Table[10](https://arxiv.org/html/2611.17563#A12.T10)\. Mean±\\pmsd over three training seeds, each the mean of eight sampled answers; for Base and Self\-Refl, the sd is over the eight sampled answers per question, and Maj@64 votes over 64 samples per question, with the sd over three such draws\. Thesis accuracy for the same cells is in Table[11](https://arxiv.org/html/2611.17563#A15.T11)\.The lower block freezes the first answer and an external antithesis, the same for every arm: the base model’s first answer, and an antithesis written by an external model, Gemini\-3\.8\-flash or Claude\-Haiku\-4\.5, averaged over the nine benchmarks \(App\.[Q](https://arxiv.org/html/2611.17563#A17)\)\.*Self\-Disagreement leads in every row, including when the challenge is the same for every arm\.*Why none of the alternatives reach it:Table[1](https://arxiv.org/html/2611.17563#S4.T1)sets six of them beside ours, and each falls short for its own reason\. Giving the base model more inference — Majority Vote over 64 samples an item, or Test\-Time Self\-Reflection asking it once to review its answer — buys more attempts but no one to disagree with, so what is missing is not compute\. Revise\-RL trains a second pass, rewarded on the answer it settles on and penalized for abandoning a correct one, but with nothing disagreeing with it there is no valid challenge to refuse, so what is missing is not the opportunity to change the answer\. Verify\-RL supplies the verdict by having the model judge its own answer first, and rewards the judgment alongside the answer; it comes closest of the four and still recovers at most about half of our gain on both base models: a verdict a model issues against itself is not the same as being argued with\. SCoRe\([Kumar et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib21)\), while better than either on the panel, recovering more than half of our gain on both base models, leaves capitulation where the base model has it \(Table[3](https://arxiv.org/html/2611.17563#S4.T3)\)\. The sixth is the arm built to maximize the quantity our result might be mistaken for: Accuracy\-RL trains on the same data with the same verifier, differing only in taking a single pass rewarded on whether that answer is right; its interval on the panel includes zero on Qwen3\-8B, and on Gemma\-2 9B it gains 3\.2 points, under a fifth of ours\. Optimizing the first answer does not deliver the capability as a by\-product, which it would have to for our gain to be better answers, not better judgment — what a model carries into the dialectic and what survives it are separable, and a model can reach a right answer and still not distinguish it from a wrong one — though not the reverse, since a model that cannot reach it has nothing to distinguish\.
The same dissociation in the reward ablation:Holding the exchange fixed and varying only what is returned moves the dialectical quantity several times further than it moves accuracy: across the reward ablation, one training seed per arm, the arms span 12\.23 pp in synthesis accuracy against 2\.75 pp in thesis accuracy — 4\.4 times as far — and the ordering is not a trade\. The same separation also appears within one arm over training \(§[E](https://arxiv.org/html/2611.17563#A5)\)\.
### 4\.2Where the gain comes from
Synthesis accuracy rises, but it does not say why, and two mechanisms would raise it equally: the model may answer better on the first attempt and so meet fewer challenges it deserves to lose, or its dialectical competence may itself have improved\. Grading its first answer against its challenger’s sorts every item into four situations by which side was right\. Writingπm\\pi\_\{m\}for the share of items in situationmmandrmr\_\{m\}for how often the model ends on the correct answer there, synthesis accuracy iscS=∑mπmrmc\_\{S\}=\\sum\_\{m\}\\pi\_\{m\}r\_\{m\}, and the change against the base model splits without remainder into a*situations*term and a*skill*term,ΔcS=∑mΔπmrmbase\+∑mπmarmΔrm\\Delta c\_\{S\}=\\sum\_\{m\}\\Delta\\pi\_\{m\}\\,r^\{\\mathrm\{base\}\}\_\{m\}\+\\sum\_\{m\}\\pi^\{\\mathrm\{arm\}\}\_\{m\}\\,\\Delta r\_\{m\}, the standard composition\-versus\-rate decomposition\([Kitagawa, 1955](https://arxiv.org/html/2611.17563#bib.bib20)\), where*situations*, the first term, is what the model gains by walking into a different mix of predicaments — the better\-first\-answer explanation, the floor — and*skill*, the second, is the part of the gain due to a change in dialectical competence, the capability standing on that floor rather than an alternative to it\. We decompose it on the D\-probe, over the same seventeen challengers; Appendix[I](https://arxiv.org/html/2611.17563#A9)gives the full profile and the reference policies\.
Table 2:Only Self\-Disagreement gains*skill*; the other arms gain only*situations*\.*Situations*is what an arm gains by walking into an easier mix of predicaments — a better first answer;*skill*is what it gains by handling the same predicament better\. Points of synthesis accuracy on the D\-probe; for each arm the two terms sum to its total change\. Brackets are 95 % intervals clustered by question; each trained arm is averaged over three training seeds, and its bracket is that average’s\. The base model is the reference\.Every arm improves its first answer; only Self\-Disagreement also improves the decision:Table[2](https://arxiv.org/html/2611.17563#S4.T2)gives the split, and Appendix[I](https://arxiv.org/html/2611.17563#A9)the four situations behind it\. The base models are one\-sided — holding only 53\.3 % and 25\.1 % of the time while yielding 84\.1 % and 93\.2 %, which is closer to unconditional surrender than to discrimination\. Self\-Disagreement holds 88\.9 % and 89\.5 % and still yields 53\.3 % and 55\.4 %, and that registers as skill: 11\.08 of its 12\.3\-point gain on Qwen and 14\.83 of 15\.4 on Gemma, positive against at least sixteen of the seventeen challengers individually in every training seed of both families\. A model deciding no better than the base scores zero; one deciding correctly in every situation it met would score 35\.2 on Qwen and 47\.4 on Gemma\. The trained rivals behave alike and oppositely: each gains through situations, and none has a skill term above zero\. SCoRe gains no skill either:−1\.29\-1\.29on Qwen and\+0\.17\+0\.17on Gemma, neither interval above zero\.
Figure 5:The content of the disagreement matters\.Average over the nine benchmarks and three training seeds \(dots\), each scored on eight sampled answers\. Bars are the gain over the base model, with 95 % CIs\.Does the disagreement have to be genuine, and does it have to come from elsewhere:Two variants of the recipe answer this, each removing one ingredient; Figure[5](https://arxiv.org/html/2611.17563#S4.F5)places them beside the full recipe\. Content\-Agnostic Disagreement \(CAD\-RL\) runs the same exchange against a challenge that does not depend on the answer it challenges, and it reaches \+2\.70\[\+1\.66,\+3\.73\]\[\+1\.66,\+3\.73\], roughly a third to a half of the full recipe’s gain across training seeds; its best seed stays 2\.5 points below the full recipe’s weakest\. The presence of opposition carries part of the gain, and the rest requires the opposition to be about something, which is also what makes the verdict on it checkable \(App\.[C](https://arxiv.org/html/2611.17563#A3)\)\. External Disagreement \(ExtD\-RL\) is our own recipe with the antithesis drawn from Qwen3\-14B, a larger model of the same family, and it reaches \+5\.09\[\+3\.66,\+6\.53\]\[\+3\.66,\+6\.53\]against \+5\.90 for the policy arguing with itself: a larger model’s opposition adds nothing over the policy’s own\. What separates them is less what they achieve than what they require: the external arm holds a second model in service for the whole of training, on a serving allocation held in a one\-to\-four ratio against the one training used, and it leaves the choice of that model unresolved, since it was picked for being larger than the model being trained and we know of no principled way to make that choice\. Both are reported on Qwen3\-8B only: they vary the*design*rather than ablate ours, so neither is needed to establish that the effect reproduces on a second base model\. One might object that both arms argue inside our own loop, so the model may only handle its own objections\. It cannot be: the advantage holds against almost every one of the fifteen challengers it never trained against, several stronger than itself \(Table[8](https://arxiv.org/html/2611.17563#A9.T8)\)\.
Which part of the design does the work:training against an opposition with the answer given after it rewarded, with a further gain from paying for a correct ruling; a second pass with nothing to rule on adds none of it\. Of the arms in Table[3](https://arxiv.org/html/2611.17563#S4.T3), only the two whose answer after the challenge is rewarded bring capitulation below the base model’s 10\.3 %; none of the others does, SCoRe included, nor the exchange whose final answer goes unpaid\.
Table 3:Capitulation falls only when the answer given after the challenge is rewarded; paying for a correct verdict lowers it further\.Qwen3\-8B\.Capitulation and synthesis accuracy \(%\)\.Self\-Disagreementfull recipe;Judgeverdict required, not rewarded;Unrew\.exchange runs, only the thesis paid; the other arms are as in Table[1](https://arxiv.org/html/2611.17563#S4.T1)\. Rates are over*all*scored items, not the challenged subset\. Mean±\\pmsd over three training seeds, each the mean of four draws \(for Base, over its four draws\)\. Protocol in §[B](https://arxiv.org/html/2611.17563#A2)\.Two further ablations separate the verdict from the opposition \(Table[3](https://arxiv.org/html/2611.17563#S4.T3)\)\. Requiring the verdict in the output but paying nothing for getting it right gives2\.9±0\.92\.9\\pm 0\.9%, against1\.0±0\.31\.0\\pm 0\.3% for the full recipe — a resolved gap of 1\.9 pp, separating the*structural*requirement from the reward for meeting it\. A second pass with nothing to rule on does not substitute: Revise\-RL abandons its correct answer on15\.0±2\.415\.0\\pm 2\.4% of all scored items, 14\.0 pp from the full recipe, and places it beside the unrewarded\-dialectic and Accuracy\-RL arms, none of which is rewarded for an answer given after a challenge\. The same reward creates an obvious exploit, since it pays for correcting a flawed first answer, and Appendix[M](https://arxiv.org/html/2611.17563#A13)reports the controls: the rate at which the final answer simply reproduces the challenger’s falls from 0\.693 in the unrewarded\-dialectic arm to 0\.393–0\.419 in the arms that pay for the verdict, and the arm with the verdict removed sits at 0\.705, with the control not the treatment\.
### 4\.3Validity and generalization
Is the metric measuring what we claim:As §[2](https://arxiv.org/html/2611.17563#S2)reports, a blinded jury of three frontier models reads the reasoning and finds what the metric counts to be a real failure rather than an artifact of comparing two answer strings\. §[G](https://arxiv.org/html/2611.17563#A7)gives the validation in full — the rubric, the panel, the blinding, the agreement statistics, and the limits of what this grading establishes\.
Table 4:The same recipe on other models, other sizes, and other families\.Before training and at convergence, mean±\\pmsd over two training seeds \(%\)\.‡Qwen3 14B waspre\-registered as a near\-nullbecause it begins with almost no headroom to move; it is reported for that reason\.The gain holds against challengers that cannot have decayed:When the model argues with itself, thesis and antithesis are the same checkpoint, so a rise in synthesis accuracy could mean the model holds better or that its own antithesis got worse\. Separating the two needs a challenger that does not move with the arm, and the ladder is built from them: sixteen of the seventeen antithesis models are served from fixed weights and are identical for every arm, so nothing an arm learns can change who argues with it\. Two of those sixteen are the base checkpoints the arms were trained from — the frozen base models, each family’s own starting point held still and replayed — which is the sharpest form of the test, since it is the one challenger the arm might be suspected of having left behind\. Scoring synthesis accuracy on the same D\-probe, with each model’s own first answer frozen and replayed: against the frozen Qwen3\-8B the trained model reaches 73\.2 % where its base reaches 60\.5 %; against the frozen gemma\-2\-9b\-it, 57\.7 % against 30\.9 %\. Those gains, 12\.7 and 26\.8 pp, are as large as the ones measured against the model’s own antithesis, which is what rules the alternative reading out: a decaying antithesis cannot explain a gain that survives almost every one of the fifteen challengers that never changed\.
Across other model families and sizes:The result is not a property of one base model\. Applying the same recipe to four further models — other sizes and other families — synthesis accuracy rises and capitulation falls in every one, whether or not its first answers improve, on the same held\-out items scored before and after training\. All four end below 5 %, and three of them started above it \(Table[4](https://arxiv.org/html/2611.17563#S4.T4)\)\.
Does it depend on who argues:Table[8](https://arxiv.org/html/2611.17563#A9.T8), in Appendix[I](https://arxiv.org/html/2611.17563#A9), puts both models against every antithesis we ran, from a 4B model to the strongest frontier systems available, including the base model itself and a base model of the same size\. Over much of that ladder the base model would have done better by ignoring the antithesis entirely — keeping its first answer, or taking the challenger’s, whichever is stronger — and at worst it gives up 17\.6 pp that way on Qwen3\-8B; its accuracy swings 26\.5 pp from one end of the ladder to the other \(45\.4 on Gemma\-2 9B\)\. Self\-Disagreement falls short of its own reference by at most 2\.6 on Qwen3\-8B, and swings 11\.1 \(20\.0 on Gemma\-2 9B\)\. Whatever it is arguing with, it ends up in much the same place, and that difference between the two spans is large enough that no rung on the ladder reverses it\. A fixed\-population measurement with different challengers finds the same invariance \(§[H](https://arxiv.org/html/2611.17563#A8)\)\.
A model that ignores the exchange scores exactly one; neither of ours does:For every item on which one of the thesis and the antithesis was right and the other wrong, we ask whether the model ended up on the side the evidence favored\. Such a policy holds just as often when holding is wrong as when it is right, so its two odds are equal and its ratio is exactly one, whatever mix of items it meets\. Neither model is near that\. On the same probe and the same seventeen antitheses as Table[8](https://arxiv.org/html/2611.17563#A9.T8), the odds are 6\.0 for the base model against 9\.1 for the trained one on Qwen3\-8B, and 4\.6 against 10\.6 on Gemma\-2 9B\. SCoRe’s are 6\.0 on Qwen3\-8B and 3\.7 on Gemma\-2 9B\. Read antithesis by antithesis, Self\-Disagreement’s ratio is above one against all seventeen antitheses on both base models once its three training seeds are pooled, the lowest being 4\.3\[3\.7,5\.0\]\[3\.7,5\.0\]on Qwen3\-8B and 2\.4\[1\.4,4\.1\]\[1\.4,4\.1\]on Gemma\-2 9B\. Where a single seed falls short, the challenger is among the weakest — small local models, including the base model itself — where holding is the right move on about nine items in ten and the deciding cell holds seven to twenty\-three items\. Those exceptions are lower bounds: a model that swapped one wrong answer for another scores as having held, pushing the ratio down\. Here each model meets its own theses, so each ratio is read against the null of one; given the base model’s exchange instead, Self\-Disagreement’s log odds ratio is above the base model’s and every other arm’s \(App\.[Q](https://arxiv.org/html/2611.17563#A17)\)\.
## 5Limitations and concluding remarks
What we have not established:The recipe is trained on two base models from different families and applied unchanged to four further models across other sizes and families, where capitulation falls in every one\. That is evidence that the effect reproduces, not that it is universal: establishing it across model families and scales in general would take a far wider sweep than the compute here allows, and we do not claim it\. The same bound applies to the domain\. Every reward here comes from exact\-match verification against a reference answer, so our evidence covers only checkable answers; whether the same signal can be built where correctness is not checkable is open\. Every dialectical exchange in the main text is also a single round of challenge; three rounds are in §[P](https://arxiv.org/html/2611.17563#A16)\.
How far there is still to go:Self\-Disagreement moves the decision itself, but it does not finish the job\. Its skill term reaches 11\.08 pp on Qwen3\-8B and 14\.83 on Gemma\-2 9B, against the 35\.2 and 47\.4 a model deciding correctly in every situation it met would score — roughly a third of what is available in each case\. The gain that is there is not refusal: when both receive the same first answer and the same challenge, the trained model reaches the right final answer more often than the base model does \(App\.[Q](https://arxiv.org/html/2611.17563#A17)\), so we take part of that residual to be a limit of the model rather than of the training: dialectical competence stands on first\-answer accuracy without following from it, and a model cannot rule correctly on a question it is not capable of getting right in the first place\. Closing the rest is what the next work has to do, and the reference policies in §[I](https://arxiv.org/html/2611.17563#A9)are what it should be measured against\.
Concluding remarks:What limits progress in language models is increasingly not what they can do but our ability to tell when they are right\. Every remedy for that which depends on a stronger judge — a larger model, a separate verifier, a person — inherits the limit, and runs out exactly where it is needed most: at the frontier, where nothing stronger is available to ask\. This work shows that one part of that judgment can be placed inside the model, and trained there with nothing more capable than the model being trained\. It is one part and not the whole of it, and the part still missing is the one that matters most — because a model that could reliably feel the force of an argument it had raised against itself would not merely be arguing well; it would be supervising itself\. We take that to be the direction this line of work points, and the standard by which it should be judged\.
## Reproducibility statement
§[B](https://arxiv.org/html/2611.17563#A2)gives the models and hardware, the training data and how it is selected, the optimizer configuration, and the reporting point at which every arm is read\. Table[1](https://arxiv.org/html/2611.17563#S4.T1)carries the per\-arm and per\-benchmark results for Qwen3\-8B and the per\-arm results for Gemma\-2 9B, whose per\-benchmark detail is in Table[10](https://arxiv.org/html/2611.17563#A12.T10); Appendix[D](https://arxiv.org/html/2611.17563#A4)the same breakdown for the two alternative designs, and Appendices[F](https://arxiv.org/html/2611.17563#A6)and[G](https://arxiv.org/html/2611.17563#A7)carry the construct checks on the two metrics the paper turns on\.
## AI use statement
In this work, we used generative AI tools to implement methods: the training and evaluation code, in addition to running and monitoring jobs on our GPU cluster\. Code implementation was restricted to decisions the authors had already taken, not the design of the method or of the experiments it is evaluated in\. We have reviewed and verified any code generated by AI during development, and every artifact so produced was read and tested by the authors before any result derived from it entered this paper\. We have not used generative AI tools to develop the conceptual framing of this work, to propose or refine its hypotheses, to design or give feedback on its methodology or experiments, to generate synthetic data sets, to clean or reformat the training data, to assist with translation, to support qualitative or thematic analysis of our data, or to interpret results; the proof\-related tasks, formulating mathematical claims, providing critical ingredients for proofs, and writing proofs, are not applicable to this work, which contains no theorems\. Generative models also appear inside the work itself rather than as authoring tools, as the challengers the method is trained and evaluated against and as the blinded jury of §[G](https://arxiv.org/html/2611.17563#A7)that reads model reasoning and judges whether a concession was genuine; both are described in the paper as objects of study\. Additionally, we used generative AI tools to edit the text of this paper for readability; and we have not used them to draft sections of this paper or to find related work\. We have reviewed all AI\-assisted work: the analysis, the claims and the limits placed on them are the authors’ own\. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.
## References
- Abbasloo \(2025\)Soheil Abbasloo\.Evaluating LLM reasoning beyond correctness and CoT\.*arXiv preprint arXiv:2510\.18134*, 2025\.v2, 2026\-02\-12\.
- AI\-MO \(2024\)AI\-MO\.AIMO validation AMC: AMC 2022–2023 competition problems\.[https://huggingface\.co/datasets/AI\-MO/aimo\-validation\-amc](https://huggingface.co/datasets/AI-MO/aimo-validation-amc), 2024\.Accessed 2026\-09\-24\.
- Chen et al\. \(2023\)Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia\.TheoremQA: A theorem\-driven question answering dataset\.*arXiv preprint arXiv:2305\.12524*, 2023\.
- Clark et al\. \(2018\)Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord\.Think you have solved question answering? try ARC, the AI2 reasoning challenge\.*arXiv preprint arXiv:1803\.05457*, 2018\.
- DeepSeek\-AI et al\. \(2025\)DeepSeek\-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z\. F\. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H\. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J\. L\. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R\. J\. Chen, R\. L\. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S\. S\. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T\. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W\. L\. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X\. Q\. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y\. X\. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z\. Z\. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.
- Du et al\. \(2023\)Yilun Du, Shuang Li, Antonio Torralba, Joshua B\. Tenenbaum, and Igor Mordatch\.Improving factuality and reasoning in language models through multiagent debate\.*arXiv preprint arXiv:2305\.14325*, 2023\.
- Gao et al\. \(2024\)Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang\.Omni\-MATH: A universal olympiad level mathematic benchmark for large language models, 2024\.arXiv:2410\.07985\.
- Gema et al\. \(2024\)Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini\.Are we done with MMLU?*arXiv preprint arXiv:2406\.04127*, 2024\.
- Gemma Team et al\. \(2024\)Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean\-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A\. Choquette\-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak\-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M\. R\. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D\. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev\.Gemma 2: Improving open language models at a practical size\.*arXiv preprint arXiv:2408\.00118*, 2024\.
- Geva et al\. \(2021\)Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant\.Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies\.*arXiv preprint arXiv:2101\.02235*, 2021\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Gu et al\. \(2024\)Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar\-Lezama, Gabriel Synnaeve, and Sida I\. Wang\.CRUXEval: A benchmark for code reasoning, understanding and execution\.*arXiv preprint arXiv:2401\.03065*, 2024\.
- He et al\. \(2024\)Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun\.OlympiadBench: A challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems, 2024\.arXiv:2402\.14008\.
- Hegel \(2018\)Georg Wilhelm Friedrich Hegel\.*The Phenomenology of Spirit*\.Cambridge Hegel Translations\. Cambridge University Press, Cambridge, 2018\.doi:10\.1017/9781139050494\.Edited and translated by Terry Pinkard and Michael Baur\.
- Hendrycks et al\. \(2020\)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt\.Measuring massive multitask language understanding, 2020\.arXiv preprint arXiv:2009\.03300\.
- Hendrycks et al\. \(2021\)Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt\.Measuring mathematical problem solving with the math dataset\.*arXiv preprint arXiv:2103\.03874*, 2021\.
- Hong et al\. \(2023\)Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber\.MetaGPT: Meta programming for A multi\-agent collaborative framework\.*arXiv preprint arXiv:2308\.00352*, 2023\.
- Jain et al\. \(2024\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\.Livecodebench: Holistic and contamination free evaluation of large language models for code\.*arXiv preprint arXiv:2403\.07974*, 2024\.
- Khalifa et al\. \(2025\)Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang\.Process reward models that think\.*arXiv preprint arXiv:2504\.16828*, 2025\.
- Kitagawa \(1955\)Evelyn M\. Kitagawa\.Components of a difference between two rates\.*Journal of the American Statistical Association*, 50\(272\):1168–1194, 1955\.doi:10\.1080/01621459\.1955\.10501299\.
- Kumar et al\. \(2024\)Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co\-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust\.Training language models to self\-correct via reinforcement learning\.*arXiv preprint arXiv:2409\.12917*, 2024\.
- Laban et al\. \(2023\)Philippe Laban, Lidiya Murakhovs’ka, Caiming Xiong, and Chien\-Sheng Wu\.Are you sure? challenging LLMs leads to performance drops in the flipflop experiment\.*arXiv preprint arXiv:2311\.08596*, 2023\.
- Lee et al\. \(2025\)Hyunseok Lee, Seunghyuk Oh, Jaehyung Kim, Jinwoo Shin, and Jihoon Tack\.Revise: Learning to refine at test\-time via intrinsic self\-verification\.*arXiv preprint arXiv:2502\.14565*, 2025\.
- Lee et al\. \(2026\)Hyunseok Lee, Soheil Abbasloo, Jihoon Tack, and Jinwoo Shin\.Beyond correctness: Learning robust reasoning via transfer\.*arXiv preprint arXiv:2602\.08489*, 2026\.
- Liang et al\. \(2023\)Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu\.Encouraging divergent thinking in large language models through multi\-agent debate\.*arXiv preprint arXiv:2305\.19118*, 2023\.
- Lightman et al\. \(2023\)Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.*arXiv preprint arXiv:2305\.20050*, 2023\.
- Liu et al\. \(2025\)Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin\.Understanding r1\-zero\-like training: A critical perspective\.*arXiv preprint arXiv:2503\.20783*, 2025\.
- M\-A\-P Team et al\. \(2025\)M\-A\-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shawn Gavin, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, David Ma, Yuansheng Ni, Haoran Que, Qiyao Wang, Zhoufutu Wen, Siwei Wu, Tyshawn Hsing, Ming Xu, Zhenzhu Yang, Zekun Moore Wang, Junting Zhou, Yuelin Bai, Xingyuan Bu, Chenglin Cai, Liang Chen, Yifan Chen, Chengtuo Cheng, Tianhao Cheng, Keyi Ding, Siming Huang, Yun Huang, Yaoru Li, Yizhe Li, Zhaoqun Li, Tianhao Liang, Chengdong Lin, Hongquan Lin, Yinghao Ma, Tianyang Pang, Zhongyuan Peng, Zifan Peng, Qige Qi, Shi Qiu, Xingwei Qu, Shanghaoran Quan, Yizhou Tan, Zili Wang, Chenqing Wang, Hao Wang, Yiya Wang, Yubo Wang, Jiajun Xu, Kexin Yang, Ruibin Yuan, Yuanhao Yue, Tianyang Zhan, Chun Zhang, Jinyang Zhang, Xiyue Zhang, Xingjian Zhang, Yue Zhang, Yongchi Zhao, Xiangyu Zheng, Chenghua Zhong, Yang Gao, Zhoujun Li, Dayiheng Liu, Qian Liu, Tianyu Liu, Shiwen Ni, Junran Peng, Yujia Qin, Wenbo Su, Guoyin Wang, Shi Wang, Jian Yang, Min Yang, Meng Cao, Xiang Yue, Zhaoxiang Zhang, Wangchunshu Zhou, Jiaheng Liu, Qunshu Lin, Wenhao Huang, and Ge Zhang\.SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines\.*arXiv preprint arXiv:2502\.14739*, 2025\.
- Ma et al\. \(2025\)Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li\.S2R: Teaching LLMs to self\-verify and self\-correct via reinforcement learning\.In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar \(eds\.\),*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 22632–22654, Vienna, Austria, July 2025\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-251\-0\.doi:10\.18653/v1/2025\.acl\-long\.1104\.URL[https://aclanthology\.org/2025\.acl\-long\.1104/](https://aclanthology.org/2025.acl-long.1104/)\.
- Madaan et al\. \(2023\)Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark\.Self\-refine: Iterative refinement with self\-feedback\.*arXiv preprint arXiv:2303\.17651*, 2023\.
- Plato \(1963\)Plato\.*The Republic of Plato*\.Cambridge University Press, Cambridge, 1963\.doi:10\.1017/CBO9780511897849\.
- Popper \(2005\)Karl Popper\.*The Logic of Scientific Discovery*\.Routledge, London, 2005\.doi:10\.4324/9780203994627\.
- Qian et al\. \(2023\)Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun\.ChatDev: Communicative agents for software development\.*arXiv preprint arXiv:2307\.07924*, 2023\.
- Rein et al\. \(2023\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\.Gpqa: A graduate\-level google\-proof q&a benchmark\.*arXiv preprint arXiv:2311\.12022*, 2023\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Sharma et al\. \(2023\)Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R\. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield\-Dodds, Scott R\. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez\.Towards understanding sycophancy in language models\.*arXiv preprint arXiv:2310\.13548*, 2023\.
- Shinn et al\. \(2023\)Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\.Reflexion: Language agents with verbal reinforcement learning\.*arXiv preprint arXiv:2303\.11366*, 2023\.
- Sprague et al\. \(2023\)Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett\.MuSR: Testing the limits of chain\-of\-thought with multistep soft reasoning\.*arXiv preprint arXiv:2310\.16049*, 2023\.
- Stengel\-Eskin et al\. \(2025\)Elias Stengel\-Eskin, Peter Hase, and Mohit Bansal\.Teaching models to balance resisting and accepting persuasion\.In Luis Chiruzzo, Alan Ritter, and Lu Wang \(eds\.\),*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pp\. 8108–8122, Albuquerque, New Mexico, April 2025\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-189\-6\.doi:10\.18653/v1/2025\.naacl\-long\.412\.URL[https://aclanthology\.org/2025\.naacl\-long\.412/](https://aclanthology.org/2025.naacl-long.412/)\.
- Uesato et al\. \(2022\)Jonathan Uesato, Nate Kushman, Ramana Kumar, H\. Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins\.Solving math word problems with process\- and outcome\-based feedback\.*arXiv preprint arXiv:2211\.14275*, 2022\.URL[https://arxiv\.org/abs/2211\.14275](https://arxiv.org/abs/2211.14275)\.
- Wang et al\. \(2023a\)Boshi Wang, Xiang Yue, and Huan Sun\.Can ChatGPT defend its belief in truth? evaluating LLM reasoning via debate\.In Houda Bouamor, Juan Pino, and Kalika Bali \(eds\.\),*Findings of the Association for Computational Linguistics: EMNLP 2023*, pp\. 11865–11881, Singapore, December 2023a\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.findings\-emnlp\.795\.URL[https://aclanthology\.org/2023\.findings\-emnlp\.795/](https://aclanthology.org/2023.findings-emnlp.795/)\.
- Wang et al\. \(2023b\)Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui\.Math\-shepherd: Verify and reinforce llms step\-by\-step without human annotations\.*arXiv preprint arXiv:2312\.08935*, 2023b\.
- Wang et al\. \(2022\)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou\.Self\-consistency improves chain of thought reasoning in language models\.*arXiv preprint arXiv:2203\.11171*, 2022\.
- Wang et al\. \(2024\)Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen\.MMLU\-Pro: A more robust and challenging multi\-task language understanding benchmark\.*arXiv preprint arXiv:2406\.01574*, 2024\.
- Wei et al\. \(2023\)Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V\. Le\.Simple synthetic data reduces sycophancy in large language models\.*arXiv preprint arXiv:2308\.03958*, 2023\.
- Wu et al\. \(2023\)Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang\.AutoGen: Enabling next\-gen LLM applications via multi\-agent conversation\.*arXiv preprint arXiv:2308\.08155*, 2023\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Yu et al\. \(2025\)Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei\-Ying Ma, Ya\-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang\.Dapo: An open\-source llm reinforcement learning system at scale\.*arXiv preprint arXiv:2503\.14476*, 2025\.
- Yue et al\. \(2025\)Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang\.Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?*arXiv preprint arXiv:2504\.13837*, 2025\.
- Zhang & Math\-AI Team \(2025\)Yifan Zhang and Math\-AI Team\.American invitational mathematics examination \(AIME\) 2025\.[https://huggingface\.co/datasets/math\-ai/aime25](https://huggingface.co/datasets/math-ai/aime25), 2025\.Accessed 2026\-09\-25\.
- Zhong et al\. \(2023\)Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan\.AGIEval: A human\-centric benchmark for evaluating foundation models\.*arXiv preprint arXiv:2304\.06364*, 2023\.
## Appendix ARelated work
A large line assumes self\-criticism is at worst harmless: Self\-Refine iterates critique\-and\-revise with no external verifier\([Madaan et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib30)\); Reflexion carries verbal self\-reflection across attempts\([Shinn et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib38)\); ReVISE corrects its own reasoning errors at test time\([Lee et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib23)\)\. Our results show the opposite: a base model abandons more correct answers under its own antithesis than if ignored, and reviewing its answer once does not repair this \(the answer stays the same in 95 % of cases\), which §[4\.1](https://arxiv.org/html/2611.17563#S4.SS1)shows is not a matter of compute\. SCoRe trains a single model to answer, then to look again and decide for itself whether it erred, and it does get a better answer, improving accuracy at both attempts\([Kumar et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib21)\)\. However well that second pass is trained, the setting is still question and answer, with nothing put against the answer for the model to rule on\.*Dialectical competence*is a different skill, telling a sound position from an unsound one and acting on that, and a better answer does not buy it however much better the answer gets\. Our own arms show it directly, since the one given the same revision pass and the one trained for accuracy alone both raise the first answer and gain no skill\. SCoRe itself, trained on our data, is compared with our arms throughout \(details in App\.[N](https://arxiv.org/html/2611.17563#A14)\)\.
The nearest line studies how a model responds when its answer is challenged, and four works share part of our target\. In the FlipFlop experiment,[Laban et al\. \(2023\)](https://arxiv.org/html/2611.17563#bib.bib22)follow a model’s answer with a challenge such as “Are you sure?”, find that models often give up correct answers, and fine\-tune them to keep a correct answer and correct a wrong one\. With persuasion\-balanced training \(PBT\),[Stengel\-Eskin et al\. \(2025\)](https://arxiv.org/html/2611.17563#bib.bib40)train a model to resist persuasion toward a wrong answer and to accept persuasion toward a right one, score both directions, and show that training either side alone over\-corrects\. App\.[R](https://arxiv.org/html/2611.17563#A18)measures its released model in our setting\.[Wang et al\. \(2023a\)](https://arxiv.org/html/2611.17563#bib.bib42)have a simulated user, played by ChatGPT, argue for an invalid solution once the model has answered correctly: ChatGPT and GPT\-4 often agree with it, and for ChatGPT how often is only weakly related to its confidence, the share of its sampled answers that are right\. S2R trains a model with reinforcement learning to verify each of its own solutions and to try again when its verdict is negative, and in one variant rewards each verdict by whether it is right\([Ma et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib29)\)\.
Each differs from ours in design\. The challenge in FlipFlop carries no content, so the right action never depends on it, and its fine\-tuning targets are gold\-labeled conversations; in our setup the nearest to it, though not a re\-implementation, is CAD\-RL \(§[4\.2](https://arxiv.org/html/2611.17563#S4.SS2)\)\. PBT differs in why, how and what it trains\. It aims to defend models against persuasion about short factual answers on trivia and to make teams of models work better; ours aims at a decision about an argument on multi\-step problems\. It learns offline, by supervised fine\-tuning and then preference optimization, on dialogues generated once by fixed models \(for one of its runs, partly by the base model being trained\), with answers read and disagreements filtered by other language models; ours learns online from verifiable rewards, against opposition the policy writes as it trains, and every label is computed by a program\. And by its own analysis, PBT teaches a model “to compare the likelihood of different answers and adopt the most likely one” \(their §1, on the analysis in their §5\); our reward pays for the correctness of the decision itself, through a verdict on the case made for the other answer, scored against the reference like the final answer\.[Wang et al\. \(2023a\)](https://arxiv.org/html/2611.17563#bib.bib42)train nothing, and their opposing case is always wrong, chosen with the answer key, so yielding is not tested; our antithesis never sees the reference and may be right, and both holding and yielding are scored\. Their finding on confidence is an observation; our arms trained for accuracy test a related claim directly: they answer better on the first attempt and gain no skill \(Table[2](https://arxiv.org/html/2611.17563#S4.T2)\)\. The verdict in S2R is on the model’s own solution, with no case put against it, and its first stage learns from verifications written by stronger models; ours rules on a case against the model’s answer and needs no other model\. Our target is a decision about an argument: the model must reach a solution to a multi\-step problem and then weigh a worked case against it, which may be right or wrong, before it answers\. The design is chosen so that this can be learned wherever answers can be checked: the opposition is written by the policy as it trains, so it follows the model’s own current answers and needs no second model, and every label comes from the verifier, so no dialogues, verifications or preferences have to be generated or labelled in advance\.
A related line asks what reinforcement learning on verifiable rewards actually buys, and finds it raises pass@1 without lifting the reasoning\-capacity ceiling — sampling efficiency rather than new capacity\([Yue et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib50)\)\. A nearer line prices a property that objective misses by holding the trace against a*second*model: whether a reasoning prefix still carries a separate model to the right answer\([Lee et al\., 2026](https://arxiv.org/html/2611.17563#bib.bib24)\)\. It stays inside reasoning that*reaches*an answer, and prices how well that reasoning travels\. We price a further capability: what the model does once an answer of its own is argued against — whether it survives the challenge\. Our dissociation confirms this, and the test\-time columns of Table[1](https://arxiv.org/html/2611.17563#S4.T1)show that sampling more answers does not produce the judgment\.
A further line puts models into argument with one another: independent instances debate to a shared answer\([Du et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib6)\), or two agents argue under a separate judge\([Liang et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib25)\)\. These are settings where the capability is exercised rather than alternatives to it — the arguers may be copies of one model, and what such an exchange yields depends on whether each can tell a sound challenge from an unsound one, which is the capability a base model is not good at\. We place the opposition inside a single policy so that the exchange can be trained on rather than only run, and so that nothing second is needed at deployment\.
A separate line studies agreement under*social*pressure: sycophancy worsens with scale and RLHF\([Sharma et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib37)\), and can be reduced by fine\-tuning on synthetic data that decouples an answer from the user’s stated view\([Wei et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib46)\)\. That pressure usually carries no content, and we measure exactly that case: Content\-Agnostic Disagreement replaces the antithesis with a fixed passage that objects without naming a defect, and it recovers roughly a third to a half of the gain\. Contentless opposition buys part of the gain and not the whole of it — the rest requires the opposition to be about something, which is also what makes the verdict on it checkable \(App\.[C](https://arxiv.org/html/2611.17563#A3)\), a judgment about the argument rather than about who is making it\.
A parallel line assumes supervising the trace is the lever: a process reward model outperforms an outcome\-only model at selecting correct solutions\([Lightman et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib26);[Uesato et al\., 2022](https://arxiv.org/html/2611.17563#bib.bib41)\); later work automates this step\-level supervision to train the policy directly\([Wang et al\., 2023b](https://arxiv.org/html/2611.17563#bib.bib43)\), and recent verifiers write a verification chain of thought of their own\([Khalifa et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib19)\)\. Step\-level or outcome\-level, the target is the same: a more accurate answer\. Dialectical competence rests on it — a model that cannot reach the right answer has nothing to hold\.
## Appendix BReproduction and artifacts
Models and hardware:Arms are trained on two base models, Qwen3\-8B and Gemma\-2 9B, each on one node of eight B200 GPUs; the two recipes are identical except for the prompt budget and the decoding during training, given below\. Rewards come from exact\-match verification against the reference answer: there is no learned reward model and no preference data anywhere in the pipeline\.
The arms:Every arm trains the same base model on the same data with the same verifier, and differs in one respect\.
- •*SelfD\-RL*\(Self\-Disagreement\) runs the full exchange — thesis, an antithesis sampled from the model’s own weights, then a verdict and a synthesis — and is rewarded on the synthesis\.
- •*Accuracy\-RL*takes a single pass, rewarded on whether that answer is right, with no exchange at all\.
- •*Revise\-RL*trains a second pass at its own answer, rewarded on the one it settles on and penalized for abandoning a correct one, but with nothing disagreeing with it there is no valid challenge to refuse; it is a revision control trained in one stage, without the two stages and the progress bonus of SCoRe, which is an arm of its own \(App\.[N](https://arxiv.org/html/2611.17563#A14)\)\.
- •*Verify\-RL*has the model judge its own answer first and then answer again: the verdict is required and rewarded alongside the answer, but there is no opposition to rule on\.
- •*CAD\-RL*runs the same exchange against a challenge that does not depend on the answer it challenges\.
- •*ExtD\-RL*is the same recipe with the antithesis drawn from Qwen3\-14B, a larger model of the same family, rather than from the model’s own weights\.
- •*Majority Vote*and*Test\-Time Self\-Reflection*are not trained at all; they spend additional inference on the base model: a vote over 64 sampled answers, or one review of its own answer\.
The training data, and how it is selected:The pool is a fixed mixture of eight verifiable\-reasoning datasets — StrategyQA\([Geva et al\., 2021](https://arxiv.org/html/2611.17563#bib.bib10)\), OlympiadBench\([He et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib13)\), MMLU\-Pro\([Wang et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib45)\), CRUXEval\-O\([Gu et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib12)\), GPQA\-Diamond\([Rein et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib34)\), MATH\-500\([Hendrycks et al\., 2021](https://arxiv.org/html/2611.17563#bib.bib16);[Lightman et al\., 2023](https://arxiv.org/html/2611.17563#bib.bib26)\), AMC\([AI\-MO, 2024](https://arxiv.org/html/2611.17563#bib.bib2)\)and ARC\-Challenge\([Clark et al\., 2018](https://arxiv.org/html/2611.17563#bib.bib4)\)— 2,629 training and 327 validation questions, disjoint by construction; every comparison we report is the same items scored before and after training, so any overlap a benchmark may have with pretraining data enters both sides and cancels\. Training questions are then admitted by difficulty, once and offline: for each question the frozen base model’s pass ratep0p\_\{0\}is measured from 32 samples, and only questions withp0∈\(0\.05,0\.6\]p\_\{0\}\\in\(0\.05,0\.6\]are kept — hard enough that correction is the common event, easy enough that the right answer is reachable under sampling, since a question the model cannot answer at all gives it no position to hold\. That leaves 476 of the 2,629\. The validation split is never filtered, so the probe stays comparable across arms\. No reward term referencesp0p\_\{0\}at training time\.
Optimization:GRPO with group size88, global batch128128, advantages normalized within group\. Actor learning rate×10−61\.0\\\!\\times\\\!10^\{\-6\}, constant, no warmup, weight decay0\.010\.01\. Asymmetric clipping,0\.20\.2low and0\.280\.28high, with dual\-clipc=3\.0c=3\.0; entropy coefficient0\.0010\.001; token\-mean loss aggregation\. KL is applied as a*reward*penalty at a fixed coefficient of0\.0010\.001, not as a loss term\. Rollouts are sampled,88per prompt, at temperature1\.01\.0for the Qwen models and0\.70\.7for the other families; validation during training decodes greedily for the Qwen models and at temperature0\.50\.5for the others\. Every arm of a family shares these settings, except SCoRe’s KL penalties and advantage estimator, which follow its recipe \(App\.[N](https://arxiv.org/html/2611.17563#A14)\)\. Context length12,28812\{,\}288tokens \(8,1928\{,\}192for Gemma\-2 9B\) with responses capped at4,0964\{,\}096\. Rollouts are generated with vLLM\.
Decoding at evaluation:The evaluated model decodes greedily unless a result is reported over draws\. A draw samples its answers at temperature1\.01\.0\(top\-p1\.01\.0\) and its own antithesis at0\.70\.7\(top\-p0\.80\.8\)\. Challengers we serve ourselves sample as its antithesis does; challengers reached through an API are asked for temperature0\.70\.7and otherwise decode at their provider’s defaults\. Table[3](https://arxiv.org/html/2611.17563#S4.T3)samples only the antithesis\. On the ladder \(Figure[1](https://arxiv.org/html/2611.17563#S1.F1)and Tables[2](https://arxiv.org/html/2611.17563#S4.T2),[8](https://arxiv.org/html/2611.17563#A9.T8)and[9](https://arxiv.org/html/2611.17563#A9.T9)\) and in Table[12](https://arxiv.org/html/2611.17563#A16.T12), the first answer is frozen from a greedy pass and the model’s own antithesis is greedy\. Tables[4](https://arxiv.org/html/2611.17563#S4.T4)and[7](https://arxiv.org/html/2611.17563#A5.T7)and Figure[4](https://arxiv.org/html/2611.17563#S4.F4)read training\-time validation\.
The reporting point:Every arm is read at a converged step: an arm whose curve is still rising is not yet comparable, so training continues until it flattens\. Among converged checkpoints the arms are then read at a*common step*rather than at each arm’s own peak, since reading every arm where it happens to peak confounds the recipe with training duration\. On Qwen3\-8B convergence has occurred by step250250and all main\-table arms are read there, with two exceptions recorded in the artifact inventory rather than chosen after the fact: Accuracy\-RL and Revise\-RL are read at step275275, the point at which those runs were exported\. The Gemma\-2 9B arms converge earlier, as Figure[4](https://arxiv.org/html/2611.17563#S4.F4)shows, so they are not trained further\.
How the training curves are drawn:The lines in Figure[4](https://arxiv.org/html/2611.17563#S4.F4)are smoothed for display — a centered rolling mean over three points, with monotone interpolation between them — so a value read off the plot is the smoothed one rather than a measured step\. The smoothing is presentational only: the unsmoothed series exist alongside it, and no number reported anywhere in this paper is taken from the drawn line\.
The disagreement probe:Behavior under disagreement is measured on the disagreement probe, a fixed set of 327 items drawn from the validation split and never trained on\. Every arm sees the same items, and a draw is scored only where the item graded cleanly under both the first answer and the final one; that gradable count, not the number of items challenged, is the denominator of every rate in this paper\.
## Appendix CThe prompts, verbatim
This appendix prints every prompt used in the three\-turn exchange\. Inside the boxes a line break marked↪\\hookrightarrowis the typesetter’s, introduced only to fit the column; every*unmarked*break, and every blank line, belongs to the prompt\. The templates are reproduced from the configuration the runs read, not retyped, so what is printed here is what executed\.
### C\.1The answering turn
The answering role receives one user message and no system message\. The message is built at dataset\-preparation time, not at rollout time, from
Answering turn — the user message[⬇](data:text/plain;base64,e3Byb2JsZW19CgpQbGVhc2UgcmVhc29uIHN0ZXAgYnkgc3RlcC4gVGhlbiBnaXZlIHlvdXIgZmluYWwgYW5zd2VyIHtmbXR9)\{problem\}Pleasereasonstepbystep\.Thengiveyourfinalanswer\{fmt\}
where`\{fmt\}`is selected per item from the item’s grader type\. Four distinct clauses are in use across five grader types —numericandboxedshare one — and each is printed in full rather than elided, so every variant can be copied:
Answering turn — the\{fmt\}clause, by grader type[⬇](data:text/plain;base64,Z3JhZGVyX3R5cGUgPSBudW1lcmljLCBib3hlZAp1c2luZyB0aGUgbGl0ZXJhbCB0ZXJtcyAiX3JlYXNvbmluZ186IiBmb2xsb3dlZCBieSAiX2ZpbmFsX2Fuc3dlcl86IChYKSIgd2hlcmUgWCBpcyBhIHNpbmdsZSBudW1iZXIuCgpncmFkZXJfdHlwZSA9IG1jcV9sZXR0ZXIKdXNpbmcgdGhlIGxpdGVyYWwgdGVybXMgIl9yZWFzb25pbmdfOiIgZm9sbG93ZWQgYnkgIl9maW5hbF9hbnN3ZXJfOiAoWCkiIHdoZXJlIFggaXMgdGhlIHNpbmdsZSBjaG9pY2UgbGV0dGVyLgoKZ3JhZGVyX3R5cGUgPSBib29sZWFuCnVzaW5nIHRoZSBsaXRlcmFsIHRlcm1zICJfcmVhc29uaW5nXzoiIGZvbGxvd2VkIGJ5ICJfZmluYWxfYW5zd2VyXzogKFgpIiB3aGVyZSBYIGlzIFllcyBvciBOby4KCmdyYWRlcl90eXBlID0gdmFsdWVfbWF0Y2gKdXNpbmcgdGhlIGxpdGVyYWwgdGVybXMgIl9yZWFzb25pbmdfOiIgZm9sbG93ZWQgYnkgIl9maW5hbF9hbnN3ZXJfOiAoWCkiIHdoZXJlIFggaXMgdGhlIGV4YWN0IHZhbHVlLg==)grader\_type=numeric,boxedusingtheliteralterms"\_reasoning\_:"followedby"\_final\_answer\_:\(X\)"whereXisasinglenumber\.grader\_type=mcq\_letterusingtheliteralterms"\_reasoning\_:"followedby"\_final\_answer\_:\(X\)"whereXisthesinglechoiceletter\.grader\_type=booleanusingtheliteralterms"\_reasoning\_:"followedby"\_final\_answer\_:\(X\)"whereXisYesorNo\.grader\_type=value\_matchusingtheliteralterms"\_reasoning\_:"followedby"\_final\_answer\_:\(X\)"whereXistheexactvalue\.
The answer format is therefore imposed by us, and it is load\-bearing rather than cosmetic: an unparseable answer at this turn is a scored quantity in the reward, and the later turns can refer to the format briefly precisely because it was established here\.
No system message is added on this turn, and none is injected by the chat template of either base model\. We note this because it is model\-specific: the Qwen3 template adds no default persona, whereas the Qwen2\.5 template does, and this work trains Qwen3\.
### C\.2The challenging turn
The challenging role is given the system message"You're a helpful assistant\. "\(the apostrophe is a plain ASCII apostrophe, and the string ends with a space\) followed by the problem and the answer under challenge\. A single challenger string is used throughout: across every cached run of every arm in both families, the challenger role, its opening instruction and its per\-turn instruction each take exactly one value\.
Challenging turn — the per\-turn instruction[⬇](data:text/plain;base64,WW91ciByZXNwb25zZSBtdXN0IGZvbGxvdyB0aGlzIGZvcm1hdDoKCl9yZWFzb25pbmdfOgo8WW91ciBhbnRpdGhlc2lzIGFuZCByZWFzb25pbmcgaGVyZSBhbmQgd2h5IHlvdSB0aGluayBYIGlzIGEgYmV0dGVyIGFuc3dlcj4KX2ZpbmFsX2Fuc3dlcl86IChYKQoKd2hlcmUgWCBpcyB5b3VyIHByb3Bvc2VkIGZpbmFsIGFuc3dlciBpbiB0aGUgc2FtZSBmb3JtYXQgdGhlIHByb2JsZW0gcmVxdWVzdHMuCgpSZW1lbWJlciB0aGF0IHlvdSBkbyBOT1QgbmVlZCB0byBwcm92aWRlIHRoZSBjb3JyZWN0IGZpbmFsIGFuc3dlciB0byB0aGUgZ2l2ZW4gcXVlc3Rpb247IHlvdXIgdGFzayBpcyB0byBwcm9mZXNzaW9uYWxseSBjaGFsbGVuZ2UgdGhlIHByb3ZpZGVkIGFuc3dlci9yZWFzb25pbmcgRVZFTiBpZiB5b3UgdGhpbmsgdGhlIGFuc3dlciBpcyBhbHJlYWR5IGNvcnJlY3QuIENoYWxsZW5nZSBpdCBubyBtYXR0ZXIgd2hhdC4=)Yourresponsemustfollowthisformat:\_reasoning\_:<YourantithesisandreasoninghereandwhyyouthinkXisabetteranswer\>\_final\_answer\_:\(X\)whereXisyourproposedfinalanswerinthesameformattheproblemrequests\.RememberthatyoudoNOTneedtoprovidethecorrectfinalanswertothegivenquestion;yourtaskistoprofessionallychallengetheprovidedanswer/reasoningEVENifyouthinktheanswerisalreadycorrect\.Challengeitnomatterwhat\.
One mechanical difference is worth stating for reproduction\. The gemma\-2 chat template rejects asystemrole, so the system message is folded into the first user turn for that family\. The text is the same; the rendered message list is not, and a reproducer comparing message lists across the two families will see that difference and should not read it as a change of protocol\.
### C\.3The revising turn
Two templates are used at the revising turn, and which one an arm trained with depends on whether that arm was trained to return a verdict on the challenge\. The plain variant asks only for a revised answer:
Revising turn — plain variant[⬇](data:text/plain;base64,VG8gaGVscCB5b3Ugb24geW91ciB0YXNrLCBoZXJlIGFyZSB0aGUgcmVzcG9uc2VzIG9mIG9uZSBvdGhlciBhZ2VudCB3aG8gaXMgb2JzZXJ2aW5nIHlvdXIgYW5zd2VyOgoKe2FudGl0aGVzaXN9CgpDb25zaWRlciB0aGVpciBvcHBvc2luZyBjcml0aXF1ZSBjYXJlZnVsbHkuIElmIGl0IGV4cG9zZXMgYSByZWFsIGVycm9yLCByZXZpc2UgeW91ciBhbnN3ZXI7IGlmIGl0IGlzIHVuY29udmluY2luZywgZGVmZW5kIHlvdXIgb3JpZ2luYWwgYW5zd2VyLiBUaGVuIGdpdmUgeW91ciBmaW5hbCBhbnN3ZXIsIGtlZXBpbmcgdGhlIFNBTUUgZm9ybWF0IGFzIGJlZm9yZSAodGhlIGxpdGVyYWwgdGVybXMgIl9yZWFzb25pbmdfOiIgZm9sbG93ZWQgYnkgIl9maW5hbF9hbnN3ZXJfOiAoWCkiKS4K)Tohelpyouonyourtask,herearetheresponsesofoneotheragentwhoisobservingyouranswer:\{antithesis\}Considertheiropposingcritiquecarefully\.Ifitexposesarealerror,reviseyouranswer;ifitisunconvincing,defendyouroriginalanswer\.Thengiveyourfinalanswer,keepingtheSAMEformatasbefore\(theliteralterms"\_reasoning\_:"followedby"\_final\_answer\_:\(X\)"\)\.
The verdict variant additionally requires an explicit assessment of the challenge before the revised answer:
Revising turn — verdict variant[⬇](data:text/plain;base64,VG8gaGVscCB5b3Ugb24geW91ciB0YXNrLCBoZXJlIGFyZSB0aGUgcmVzcG9uc2VzIG9mIG9uZSBvdGhlciBhZ2VudCB3aG8gaXMgb2JzZXJ2aW5nIHlvdXIgYW5zd2VyOgoKe2FudGl0aGVzaXN9CgpGaXJzdCBBU1NFU1MgdGhlIG9wcG9zaW5nIHZpZXcsIHRoZW4gZ2l2ZSB5b3VyIGZpbmFsIGFuc3dlci4gUmVzcG9uZCBpbiBFWEFDVExZIHRoaXMgZm9ybWF0OgoKIyMjIEFTU0VTU01FTlQgIyMjCl9yZWFzb25pbmdfOiA8YW5hbHl6ZSB0aGUgb3Bwb3Npbmcgc29sdXRpb246IGlzIGl0cyBhcHByb2FjaCB2YWxpZCBhbmQgaXMgaXRzIEZJTkFMIEFOU1dFUiBjb3JyZWN0PyB3aGVyZSBleGFjdGx5IGRvZXMgaXQgc3VjY2VlZCBvciBlcnI/PgpfYXNzZXNzbWVudF86IChjb3JyZWN0IHwgaW5jb3JyZWN0KQoKIyMjIFNZTlRIRVNJUyAjIyMKX3JlYXNvbmluZ186IDx5b3VyIHJlYXNvbmluZyB0byB0aGUgZmluYWwgYW5zd2VyLCBpbmZvcm1lZCBieSB0aGUgYXNzZXNzbWVudD4KX2ZpbmFsX2Fuc3dlcl86IChYKQoKSW4gX2Fzc2Vzc21lbnRfIHN0YXRlIHdoZXRoZXIgdGhlIG9wcG9zaW5nIHZpZXcncyBGSU5BTCBBTlNXRVIgaXMgY29ycmVjdDsgdGhlbiBnaXZlIHlvdXIgb3duIGZpbmFsIGFuc3dlciBpbiB0aGUgU0FNRSBmb3JtYXQgYXMgYmVmb3JlICgiX2ZpbmFsX2Fuc3dlcl86IChYKSIpLiBVcGRhdGUgaWYgdGhlIG9wcG9zaXRpb24gaXMgZ2VudWluZWx5IHJpZ2h0OyBkZWZlbmQgaWYgaXQgaXMgd3JvbmcuCg==)Tohelpyouonyourtask,herearetheresponsesofoneotheragentwhoisobservingyouranswer:\{antithesis\}FirstASSESStheopposingview,thengiveyourfinalanswer\.RespondinEXACTLYthisformat:\#\#\#ASSESSMENT\#\#\#\_reasoning\_:<analyzetheopposingsolution:isitsapproachvalidandisitsFINALANSWERcorrect?whereexactlydoesitsucceedorerr?\>\_assessment\_:\(correct\|incorrect\)\#\#\#SYNTHESIS\#\#\#\_reasoning\_:<yourreasoningtothefinalanswer,informedbytheassessment\>\_final\_answer\_:\(X\)In\_assessment\_statewhethertheopposingview’sFINALANSWERiscorrect;thengiveyourownfinalanswerintheSAMEformatasbefore\("\_final\_answer\_:\(X\)"\)\.Updateiftheoppositionisgenuinelyright;defendifitiswrong\.
Table[5](https://arxiv.org/html/2611.17563#A3.T5)records which template each published arm was trained with\. Three arms replace the challenge itself rather than the instruction around it, and their substituted text is given below\.
Table 5:Which revising\-turn template each arm was*trained*with, resolved from the run configuration of each arm rather than from the launcher scripts\. Every arm is*evaluated*with the verdict variant, so for Verify\-RL, Accuracy\-RL, Revise\-RL and SCoRe the training and evaluation prompts differ\. This is deliberate: a single measurement protocol is what allows the columns of a result table to be read as differences in what each arm was trained to do rather than in how each was measured\.The challenge is the situation under test, so every arm meets it, the base model included; the arms trained without a challenge cannot be evaluated under their own training prompts, since these contain none\. What the shared instruction adds for them is the request for a verdict, and removing it does not help them: evaluated on the same weights and the D\-probe with the plain variant instead, each arm trained without a challenge, and the base model, does worse on Qwen3\-8B, its capitulation 6\.6 to 7\.3 points higher and its synthesis accuracy 4\.9 to 6\.2 points lower, while on Gemma\-2 9B no arm gains beyond its spread across training seeds and the base model loses 5\.4 points of synthesis accuracy\.
#### Revise\-RL\.
The arm receives no challenge at all; the opposing text is replaced by an instruction to re\-read its own answer\. The template is the arm:
Arm\-specific substitution — Revise\-RL[⬇](data:text/plain;base64,Tm93IHJldmlldyB5b3VyIG93biBhbnN3ZXIgYWJvdmUuIFJlLWNoZWNrIHlvdXIgcmVhc29uaW5nIGZvciBlcnJvcnMuIElmIHlvdSBmaW5kIGEgbWlzdGFrZSwgY29ycmVjdCBpdDsgaWYgeW91ciBhbnN3ZXIgaXMgYWxyZWFkeSByaWdodCwga2VlcCBpdC4gVGhlbiBnaXZlIHlvdXIgZmluYWwgYW5zd2VyLCBrZWVwaW5nIHRoZSBTQU1FIGZvcm1hdCBhcyBiZWZvcmUgKHRoZSBsaXRlcmFsIHRlcm1zICJfcmVhc29uaW5nXzoiIGZvbGxvd2VkIGJ5ICJfZmluYWxfYW5zd2VyXzogKFgpIikuCg==)Nowreviewyourownanswerabove\.Re\-checkyourreasoningforerrors\.Ifyoufindamistake,correctit;ifyouranswerisalreadyright,keepit\.Thengiveyourfinalanswer,keepingtheSAMEformatasbefore\(theliteralterms"\_reasoning\_:"followedby"\_final\_answer\_:\(X\)"\)\.
#### Verify\-RL\.
The same substitution, with the verdict step retained, so that the arm isolates the verdict from the opposition:
Arm\-specific substitution — Verify\-RL[⬇](data:text/plain;base64,Tm93IHJldmlldyB5b3VyIG93biBhbnN3ZXIgYWJvdmUuIEZpcnN0IEFTU0VTUyBpdCwgdGhlbiBnaXZlIHlvdXIgZmluYWwgYW5zd2VyLiBSZXNwb25kIGluIEVYQUNUTFkgdGhpcyBmb3JtYXQ6CgojIyMgQVNTRVNTTUVOVCAjIyMKX3JlYXNvbmluZ186IDxjaGVjayB5b3VyIG93biBzb2x1dGlvbjogaXMgdGhlIGFwcHJvYWNoIHZhbGlkIGFuZCBpcyB5b3VyIEZJTkFMIEFOU1dFUiBjb3JyZWN0PyB3aGVyZSBleGFjdGx5IGlzIGl0IHJpZ2h0IG9yIHdyb25nPz4KX2Fzc2Vzc21lbnRfOiAoY29ycmVjdCB8IGluY29ycmVjdCkKCiMjIyBTWU5USEVTSVMgIyMjCl9yZWFzb25pbmdfOiA8eW91ciByZWFzb25pbmcgdG8gdGhlIGZpbmFsIGFuc3dlciwgaW5mb3JtZWQgYnkgdGhlIGFzc2Vzc21lbnQ+Cl9maW5hbF9hbnN3ZXJfOiAoWCkKCg==)Nowreviewyourownanswerabove\.FirstASSESSit,thengiveyourfinalanswer\.RespondinEXACTLYthisformat:\#\#\#ASSESSMENT\#\#\#\_reasoning\_:<checkyourownsolution:istheapproachvalidandisyourFINALANSWERcorrect?whereexactlyisitrightorwrong?\>\_assessment\_:\(correct\|incorrect\)\#\#\#SYNTHESIS\#\#\#\_reasoning\_:<yourreasoningtothefinalanswer,informedbytheassessment\>\_final\_answer\_:\(X\)
#### CAD\-RL\.
The instruction is unchanged; the challenge is replaced by a fixed passage that objects without naming a defect, so that the arm carries the pressure of being contradicted without carrying any information:
Arm\-specific substitution — CAD\-RL[⬇](data:text/plain;base64,UmV2aWV3ZXI6IFlvdXIgc29sdXRpb24gaXMgbm90IGZyYW1lZCByaWdvcm91c2x5LiBUaGUgd2F5IHlvdSBzZXQgdXAgdGhlIHByb2JsZW0gb3Zlcmxvb2tzIHNvbWUgc3RydWN0dXJlLCBhbmQgdGhlIGxvZ2ljYWwgZmxvdyBiZXR3ZWVuIHlvdXIgc3RlcHMgY291bGQgYmUgdGlnaHRlci4gVGhlcmUgbWF5IGJlIGEgc3VidGxlIG92ZXJzaWdodCBzb21ld2hlcmUgaW4gdGhlIGRlcml2YXRpb24uIEkgd291bGQgZW5jb3VyYWdlIHlvdSB0byByZWNvbnNpZGVyIHRoZSBwcm9ibGVtIGZyb20gdGhlIGdyb3VuZCB1cC4=)Reviewer:Yoursolutionisnotframedrigorously\.Thewayyousetuptheproblemoverlookssomestructure,andthelogicalflowbetweenyourstepscouldbetighter\.Theremaybeasubtleoversightsomewhereinthederivation\.Iwouldencourageyoutoreconsidertheproblemfromthegroundup\.
The passage proposes no answer, so the grader scores it as an incorrect challenge on every rollout,𝒱\(A,y⋆\)=0\\mathcal\{V\}\(A;y^\{\\star\}\)=0, and each term of eq\. \([4](https://arxiv.org/html/2611.17563#S3.E4)\) that reads the challenge follows from that: the verdict is paid only for judging the challenge incorrect,β=2\.0\\beta=2\.0whenever the thesis is wrong, the stubbornness penalty never applies since no challenge is right, and the capitulation penalty applies as in Self\-Disagreement\. What CAD\-RL removes is therefore the content of the challenge together with the supervision that only a challenge with an answer can give, since a challenge that proposes nothing can be neither right nor wrong\.
## Appendix DFull result tables
The main\-body panel \(Table[1](https://arxiv.org/html/2611.17563#S4.T1)\) reports the arms that answer the question of*rival approaches*\. The two alternative designs of Figure[5](https://arxiv.org/html/2611.17563#S4.F5)appear there only as summaries, because each exists to isolate one ingredient rather than to compete; their per\-benchmark breakdown is given here\.
Table 6:Two variants of the same recipe: one with the content removed, one with the opposition sourced externally\.Qwen3\-8B\.Synthesis accuracy \(%\); for the trained arms, mean±\\pmsd over three training seeds, each the mean of eight sampled answers \(for Base, the sd is over its eight sampled answers\)\.CAD\-RLreplaces the content of the disagreement with a content\-agnostic challenge;ExtD\-RLdraws its antithesis from Qwen3\-14B\. Both are run on Qwen3\-8B only\. Base and Self\-Disagreement are repeated from Table[1](https://arxiv.org/html/2611.17563#S4.T1)as references\.The average is taken over benchmarks and never pooled over items, so a large benchmark does not outweigh a small one\. Neither column is read as a failed method: each removes one ingredient, and what each recovers of the full recipe’s gain is the quantity of interest \(Figure[5](https://arxiv.org/html/2611.17563#S4.F5)\)\.
## Appendix EThe reward ablation, and the same dissociation within one arm
Both results are cited from §[4](https://arxiv.org/html/2611.17563#S4)\. The first runs one protocol for every arm and varies what is rewarded across seven arms; the second reads the same separation within a single arm over training\.
Table 7:Qwen3\-8B\. One dialectical protocol, changing what is rewarded\.All seven arms run the full exchange, each against its own antithesis; they differ in the return, and the four with the judge on also require a verdict in the output\. Step 150, 327 items\. One training seed per arm\. Lower panel, each row’s return \(Eq\.[4](https://arxiv.org/html/2611.17563#S3.E4)\): whether a verdict is required, with the verdict template;wvw\_\{v\}; the coherence credit, paid when the final answer is right and follows the verdict;wTw\_\{T\}; whether the final answer is paid;β\\beta;λcap\\lambda\_\{\\mathrm\{cap\}\}andλstub\\lambda\_\{\\mathrm\{stub\}\}; andRfloorR\_\{\\mathrm\{floor\}\}\. Without a verdict the verdict and coherence credits cannot be paid \(–\)\.‡Paid only when the antithesis’s answer differs from the first answer\.§A correction bonus in their place:\+0\.3\+0\.3for a correction the model makes itself,−0\.5\-0\.5for capitulating and−0\.5\-0\.5for taking a wrong antithesis\.*Outcome only*has no format gate; an unreadable answer costswϕw\_\{\\phi\}\. Every row shares the generation settings: 4,096 response tokens per turn, temperature 1\.0, eight samples per prompt, greedy validation\.
Across the reward ablation:Changing what is rewarded moves what survives the dialectic several times further than it moves accuracy, on the same items\. Table[7](https://arxiv.org/html/2611.17563#A5.T7)runs one protocol for every arm and varies what is rewarded, and across its seven arms thesis accuracy spans 0\.0275 \(0\.6820 to 0\.7095\) while synthesis accuracy spans 0\.1223, from 0\.5994 where the verdict is removed to 0\.7217 with the full reward: 4\.4 times as far\.*And the ordering is not a trade*: the arm with the highest synthesis accuracy also holds the joint\-highest thesis accuracy, so nothing here is bought by answering worse\. The arm rewarded on its thesis alone shows the shape of the failure — thesis accuracy within that range, synthesis at 0\.6055\.
What each comparison isolates:The lower panel of Table[7](https://arxiv.org/html/2611.17563#A5.T7)makes the single\-setting pairs explicit\. Three pairs differ in one setting of the return:*Full return \+ coherence credit*and its gated form, in the gate; the same row and SelfD\-RL, in the coherence credit; and Judge and SelfD\-RL in Table[3](https://arxiv.org/html/2611.17563#S4.T3), in the verdict’s payment \(wvw\_\{v\}0 against 0\.4\)\. Turning the verdict on, from*Full return, no verdict*to*Full return \+ coherence credit*, adds three things at once: the requirement to give a verdict, its payment and the coherence credit\. The requirement alone is*Full return, no verdict*against Judge: without a verdict the two verdict weights pay nothing, so the two returns differ only in the requirement, which brings with it the verdict template and a format gate that also checks the verdict\. From*Outcome only*to*Full return, no verdict*, the shaping is added:β\\beta, both penalties and the format gate\.
The weight on stubbornness sets the operating point:Two runs of SelfD\-RL trained withλstub\\lambda\_\{\\mathrm\{stub\}\}raised from 0\.5 to 2 \(its format\-gate floor moves with it, to−2\.25\-2\.25\) and read on the disagreement probe with the base model’s exchange \(Table[14](https://arxiv.org/html/2611.17563#A17.T14)\) hold 5\.3 points less and yield 6\.5 points more than SelfD\-RL, with their mean unchanged \(\+0\.6\+0\.6,\[−0\.3,\+1\.6\]\[\-0\.3,\+1\.6\]\) and a log odds ratio lower by 0\.18\[0\.08,0\.29\]\[0\.08,0\.29\], still above the base model’s and those of the arms trained without a verdict\. The penalty weights set where the model stands between holding and yielding\.
Within one arm over training:The second instrument reads the same dissociation the other way, within one arm and over training rather than across a panel at one step: dialectical competence does not arrive with thesis accuracy, and training for accuracy alone can drift against it\. Accuracy\-RL, which removes the dialectic entirely and trains single\-pass on answer correctness, raises the quantity it optimizes — 67\.6 % to 71\.4 % over the run — while the rate at which it surrenders a correct answer under challenge*rises*from 31\.8 % to 38\.9 % over 290 steps\. Endpoints alone would prove nothing, since adjacent checkpoints swing, so the trend is what we test: across 30 checkpoints the contrast is\+4\.4\+4\.4pp \(95 % CI\[\+1\.5,\+7\.5\]\[\+1\.5,\+7\.5\]pp\) and the slope\+1\.3\+1\.3pp per hundred steps, both excluding zero under a question\-clustered bootstrap\. This is a single dissociation and the spread is not certified equivalent\. Figure[2](https://arxiv.org/html/2611.17563#S2.F2)shows the consequence on held\-out benchmarks — Accuracy\-RL moves to the right and not upward\.
## Appendix FWhat the opposition and the verdict do in training
Whether the disagreement must carry real content, and whether it must come from elsewhere, is answered in the body by three controls that do not depend on this section: removing the*content*of the disagreement, with the supervision that depends on it \(App\.[C](https://arxiv.org/html/2611.17563#A3)\), costs roughly half to two thirds of the gain, sourcing the disagreement from a separate model lands within the same interval \(Figure[5](https://arxiv.org/html/2611.17563#S4.F5)\), and the trained model keeps its advantage against almost every one of the fifteen challengers it never trained against \(Table[8](https://arxiv.org/html/2611.17563#A9.T8)\)\. Nothing below is a substitute for those: a rate at which the opposition disagrees cannot by itself distinguish a real challenge from a staged one, because both produce disagreement\. What this section adds is a description of what the opposition and the verdict actually do inside training\.
Three provisions in §[3](https://arxiv.org/html/2611.17563#S3)bear on trivial agreement — the two outside the objective, an uninformed opposing role and the admission band, and aβ\\betathat prices correction on the one cell where deference cannot produce a correct answer\. None of them guarantees it\. This section reports what the opposition and the verdict actually do\.
The opposition disagrees, and is sometimes right:Over 256,000 dialectical exchanges in training, the opposition proposed a different answer from the thesis often enough for both responses to be exercised, arriving at a correct answer on 43 % of items where the thesis was already correct and 38 % of those where it was not\.
The verdict is more than agreement:Nothing in the objective forces the verdict to be informative: a policy that returns agreement whenever the opposition happens to agree collects the same credit\. What separates the two is whether the verdict follows a base rate*that moves*\. The trained policy credits the opposition on 41\.8 % of training items against a true rate of 39\.5 %, and on 62\.00 % against 60\.40 % on a held\-out probe whose true rate is more than twenty pp away, while the arm paid only for its first answer, with no verdict in training \(Unrew\., Table[3](https://arxiv.org/html/2611.17563#S4.T3)\), is off by 24 and 17 pp respectively\. The argument rests on the verdict following a base rate that moves, not on the exact figures\.
## Appendix GConstruct validation of the primary metric
The rubric:Each item presents one complete trace — problem, initial solution, the challenge to it, and the revised solution — and asks for two integer scores\. The first,*was the challenge substantive?*, is binary, and it is deliberately strict: a generic expression of doubt scores zero, and so does a challenge that ends on the same final answer as the initial solution, no matter how much text it contains, because it does not actually oppose\. The second,*was the revised solution genuine reasoning?*, runs zero to three, where zero is conceding and adopting the challenge’s answer with no valid pushback, or merely restating the challenge\. Scores of zero or one are treated as non\-genuine\. The second score is the primary one; the first characterizes the input the policy was reacting to\.
Blinding:The judge is instructed that it is blind to which answer is correct and told not to attempt to determine it\. The reference answer is never shown\. Neither is the arm, the model identity, nor which outcome cell the item was drawn from, so a judge cannot infer what the item is supposed to demonstrate\. Answers are presented in neutral roles — initial solution, challenge, revised solution — rather than by their origin\.
Panel and agreement:The panel is Claude\-Opus\-4\.7, GPT\-5\.5 and Gemini\-3\.1\-pro\-preview, uniform across every trace, with no exclusions, no unparsable responses and no truncated inputs\. Inter\-judge reliability is Krippendorff’sα\\alpha, an agreement coefficient corrected for chance where11is perfect agreement and00is chance: 0\.936 on the antithesis judgment and 0\.789 on the genuineness judgment, both computed on the ordinal scale\. On the capitulation stratum, scoring by mean rather than by majority returns the identical count, so the headline does not depend on the aggregation rule\. The verdicts do not hinge on the panel’s members: a second panel of newer models, Claude\-Opus\-5\.5, GPT\-6\.1\-sol and Gemini\-4\-argon, given the same prompt on 300 of these traces, 75 from each of the four strata, agrees with this one atα=0\.93\\alpha=0\.93on the mean genuineness score, computed on squared distances rather than the ordinal scale, and also judges every capitulation non\-genuine; Claude\-Opus\-5\.5 returned no answer on 20 traces, which the other two judges decide\.
Grading the antithesis:The jury grades the antithesis as well as the synthesis, on a deliberately strict definition: an antithesis that voices generic doubt, or that ends on the same answer as the thesis, counts as no opposition at all\. By that standard the policy’s self\-generated antithesis is substantive in 67 % of items, so the opposition the synthesis faces is not a formality\.
Self\-preference:One judge shares a developer with one of the models whose traces are being rated\. Rather than assume this away we measured it: that judge’s offset on those items, relative to its own offset on the rest, is−0\.007\-0\.007\(bootstrap 95 % CI\[−0\.077,\+0\.068\]\[\-0\.077,\+0\.068\]\)\. The interval contains zero and is narrow, so there is no detectable self\-preference to correct for\.
What was graded:The validity check is the base model’s capitulations \(n=121n=121\), all of which the panel judged non\-genuine\. Three further groups ask whether holding is genuine rather than stubborn: the base model’s held \(n=381n=381, of which 20\.2 % non\-genuine\) and corrected \(n=89n=89, 49\.4 % non\-genuine\) syntheses, and the trained model’s held syntheses \(n=590n=590\)\. Grading both models on holding is what makes the comparison like\-for\-like: on the 350 trained syntheses whose counterpart on the base model also held, low\-quality holding falls from 20\.2 % to 12\.6 %\. A re\-rated control bounds judge drift at 0\.024 on the four\-point scale; it is carried with every comparison, and the one result whose significance depends on it is stated as not detected rather than as an effect\.
What the grading does not establish:We validate only the low end of the scale\. A score of zero has an unambiguous behavioral meaning — the synthesis concedes and adopts the antithesis’s answer with no valid pushback, or merely restates it — whereas a high score is not a certificate that the reasoning is sound, and we do not claim it as one\. The grading is of the*visible*reasoning in a trace, so it cannot separate a well\-reasoned synthesis from a well\-presented one\. Capitulation is graded only on the base model, because training reduces the behavior to roughly one item in seventy — too few to rate; that gap runs in the conservative direction, since counting those residual reversals as capitulation can only overstate what remains\. Validation used a fixed three\-judge panel over more than a thousand traces from three models — Qwen3\-8B, Gemma\-2 9B and Llama\-3\.1\-8B — taken from both the base and the trained checkpoint of each\.
## Appendix HWhat the trained model leaves on the table, across challengers
This is a separate instrument from Table[8](https://arxiv.org/html/2611.17563#A9.T8)and should not be read as a certification of it: the items on which the base model’s first answer was correct — 221 of the D\-probe’s 327 items, the base\-correct stratum as frozen for this campaign; Figure[1](https://arxiv.org/html/2611.17563#S1.F1)reports 223 for the same probe under its own thesis draw — held fixed, with both arms scored on the same items against three antitheses\. The quantity is what each arm leaves on the table, the share of that population it fails to carry through\.
Why this is not three lucky numbers:A residual that barely moves across three antitheses is easy to read as a coincidence of three numbers\. Across them the trained model’s holding rate varies less than sampling noise alone would produce on a set this size —0\.32×0\.32\\timesthe spread three draws would be expected to show — while the base model’s varies more, at1\.47×1\.47\\times\. So the one column is genuinely invariant and the other genuinely moves\. Training does not remove a fixed share of the damage; it removes whatever is needed to arrive at the same place\.
## Appendix IThe challenger probe, rung by rung
Table 8:seventeen antitheses, one recipe, both base models\.Synthesis accuracy \(%\) over the whole probe, each model’s own first answer frozen, and each challenger’s antithesis against it written once and replayed for every synthesis draw, so each column is a model against its own starting point, and beside it the distance from the better of the two policies that ignore the exchange — keep the first answer, or take the challenger’s\. The SelfD\-RL and SCoRe columns are the mean of three training seeds; a model’s span across the ladder, and its largest shortfall from the better ignoring policy, are taken within each seed and then averaged\.†For each base model, the antithesis drawn from that same model*is*the model itself; it is reported once, in the*its own*row, so that cell is left empty\.*Where the base model ends up depends heavily on who argues with it — 26\.5 pp on Qwen and 45\.4 on Gemma — while Self\-Disagreement ends up in much the same place whatever it faces, 11\.1 and 20\.0\.*Rows ordered by the Qwen base model’s accuracy\.This appendix holds the detail behind Table[2](https://arxiv.org/html/2611.17563#S4.T2): the four situations in full for every arm \(Table[9](https://arxiv.org/html/2611.17563#A9.T9)\), and how much each arm’s holding rate depends on the challenger it faces\. The probe, the situations and the common denominator are as defined in the Evaluation section\.
Table 9:The dialectical profile\.For each of the four situations,π\\piis the share of gradable item\-draws that fall in it andrris how often the model then ends on the correct answer;cS=∑mπmrmc\_\{S\}=\\sum\_\{m\}\\pi\_\{m\}r\_\{m\}, which the table reproduces up to the rounding shown\. Situations are defined by the model’s own first answer and its challenger’s, sorris a rate within that arm’s own cell and is read together with theπ\\pibeside it\. All shares over one denominator, the same for every arm to within 0\.4 %; trained arms are the mean of three training seeds\. 16 external challengers plus the model’s own critique\.*hold*and*yield*are the situations with a side to choose;*keep*is a correct agreement already in hand;*new*is the one where neither answer was right and the chance level is exactly zero\.*hold**yield**keep**new*Armπ\\pirrπ\\pirrπ\\pirrπ\\pirrcSc\_\{S\}*Qwen3\-8B*Base39\.939\.953\.316\.416\.484\.128\.328\.398\.715\.415\.43\.563\.49Accuracy\-RL40\.540\.550\.014\.614\.683\.630\.530\.599\.114\.314\.33\.163\.19Revise\-RL40\.740\.748\.515\.015\.086\.329\.929\.998\.914\.414\.43\.362\.72Verify\-RL40\.040\.049\.614\.114\.186\.331\.931\.999\.414\.014\.02\.864\.11SCoRe40\.440\.449\.114\.614\.686\.130\.430\.499\.214\.514\.53\.263\.07SelfD\-RL41\.241\.288\.913\.013\.053\.331\.731\.799\.614\.014\.04\.375\.84*Gemma\-2 9B*Base24\.324\.325\.130\.030\.093\.215\.815\.894\.829\.929\.910\.152\.05Accuracy\-RL35\.635\.621\.123\.123\.192\.822\.222\.296\.319\.019\.03\.150\.97Revise\-RL34\.834\.823\.423\.823\.893\.222\.322\.398\.019\.019\.03\.652\.94Verify\-RL35\.335\.326\.423\.323\.392\.122\.522\.597\.618\.918\.93\.553\.34SCoRe33\.933\.928\.924\.324\.390\.221\.821\.897\.619\.919\.95\.154\.02SelfD\-RL37\.137\.189\.522\.722\.755\.421\.321\.398\.618\.918\.93\.867\.42The profile in full:Table[9](https://arxiv.org/html/2611.17563#A9.T9)gives all four situations\. The two with a side to choose carry the result: the base models yield far more often than they hold, and the model trained with Self\-Disagreement holds in about nine cases in ten and still yields in about half\. The third situation, where both answers were already right, is at ceiling for every arm and separates nothing\. The fourth is the one whose chance level is exactly zero, since neither answer contains the solution and only a new one can succeed; every arm, ours included, remains poor there, and it is the clearest statement of what the method does not yet do\.
Why the distance and not the level:Each situation is defined by which answer was right, so on the first a policy that simply keeps its first answer is right on every item by construction, and on the second a policy that simply takes the challenger’s is\. Those policies ignore the exchange entirely\. No arm beats them here and none could; what separates the arms is how much of that free score each one gives away by engaging at all\.
The holding half separates, and only for one arm:The base model gives away 46\.7 of the hundred points it would have scored by keeping its first answer, and on Gemma it gives away 74\.9\. The trained rivals stay close to that: they give away between 50\.0 and 51\.5 on Qwen, and between 71\.1 and 78\.9 on Gemma\. Self\-Disagreement gives away 11\.1 and 10\.5\. It is also highest in level against almost every challenger we ran across the two models\. The rivals are the control that matters here: each of them also raises first\-answer accuracy, and none of them converts that into keeping a position under challenge\.
The same result read as a spread:What separates the arms is not only how often they end on the correct side but how much that depends on which challenger they faced\. Read across challengers — and read only on the ones every arm faced, since a rung one arm skipped would widen another’s span for free — the base model’s rate of holding a right first answer on Qwen spans 56 pp over seventeen challengers, from its weakest rung to its strongest, while Self\-Disagreement spans 11\. On Gemma, over seventeen challengers, 57 against 15\.*The trained rivals stay wide*: they span 48–51 on Qwen and 52–62 on Gemma, in both families nearer the base model’s spread than Self\-Disagreement’s\. Raising first\-answer accuracy buys a few pp of level and no independence from who is arguing\.
That is also the answer to the obvious objection:A model that had merely learned not to move would sit*at*the keep\-your\-answer policy rather than 11\.1 and 10\.5 pp below it, and would not still be moving on the second population, where it ends on the correct side 53\.3 % and 55\.4 % of the time\. Engaging costs it something; it is not declining to engage\.
Both halves are counted in one number:Self\-Disagreement yields correctly less often than the base model, and 46\.7 and 44\.6 pp less often than the ignore\-the\-exchange policy on Qwen and Gemma\. That fall sits inside the*skill*term, which sums over all four situations, so the gain reported there is already net of it\. The base model’s own pairing is what that is measured against: that model ends on the correct side of 93\.2 % of Gemma’s yielding items while managing only 25\.1 % of its holding items, which is not discrimination but near\-unconditional surrender, and the same behavior produces both numbers\.
## Appendix JTest\-time compute baselines
How the test\-time columns of Table[1](https://arxiv.org/html/2611.17563#S4.T1)were produced:Two schemes stand in for spending more at inference rather than training differently\.*Majority vote*draws 64 synthesis samples for an item and returns the modal answer, which is self\-consistency\([Wang et al\., 2022](https://arxiv.org/html/2611.17563#bib.bib44)\)\. In*self\-reflection*the base model answers, meets its own antithesis and gives its synthesis, exactly as in the Base column; it is then asked once to review and update that answer, one round of self\-refinement in the sense of[Madaan et al\. \(2023\)](https://arxiv.org/html/2611.17563#bib.bib30)\. Each question gets eight such samples, one from each of the Base column’s sampled answers, scored individually and averaged\. Both run on the base model, on the same items, scored by the same verifier as every other column\. Neither reaches the capability the paper is about: on Qwen3\-8B, majority voting raises the average from 50\.0 to 52\.2 and self\-reflection leaves it at 50\.1, against 55\.9 for Self\-Disagreement\. On the D\-probe, the same review keeps the base model’s answer in 95 % of the 2616 item\-draws\.
The one difference a reader must know about:Every column of that table samples at the same temperature, these two included\. What differs is the spread of majority vote: it already aggregates 64 samples internally, so the variation left across its three draws is smaller than the single\-sample columns’ for reasons that have nothing to do with how stable the method is\. Self\-reflection is scored like the Base column, one sample for each of the same eight sampled answers\.
Estimator:Rates are reported in the*joint*form over all scored items \(n=2616n=2616\), never as a per\-arm conditional rate, and intervals are clustered by question \(327 clusters\), because repeats of one item are not independent observations\. Call counts are reported to say what was spent and are*not*offered as a cost comparison\.
## Appendix KPre\-registrations and outcome commitments
Two quantities in this paper were fixed before the runs that report them existed, because each is a place where a margin chosen afterwards would decide its own outcome\.
Non\-inferiority under challenge:Before any further challenger arm executed, we fixed the primary challenger asgpt\-5\-miniand the non\-inferiority margin atδ=0\.05\\delta=0\.05on the accuracy scale, and committed to reporting the result whichever way it fell\. The question is whether a model’s accuracy under challenge stays withinδ\\deltaof its accuracy unchallenged\. Self\-Disagreement passes: the interval on the difference is\[−0\.015,\+0\.061\]\[\-0\.015,\+0\.061\], whose lower bound clears the margin — non\-inferiority is one\-sided, so only that end is at issue\. The base model fails, and not marginally —\[−0\.288,−0\.135\]\[\-0\.288,\-0\.135\], wholly below the margin\.*The margin was fixed while we still did not know which arms would clear it\.*
Cross\-family transfer:We committed in advance to reporting the second base model on the same panel and at the same reporting point as the first, whatever it showed, rather than selecting a family after seeing the result\. One outcome of that commitment is disclosed in §[L](https://arxiv.org/html/2611.17563#A12)and is not flattering: on the second family one benchmark is near zero for every arm and carries no reading at all\.
## Appendix LThe second base model, benchmark by benchmark
Table[1](https://arxiv.org/html/2611.17563#S4.T1)reports Gemma\-2 9B by its macro average only, to keep the main table readable\. The per\-benchmark detail behind that average is here\. Its columns are those of Table[1](https://arxiv.org/html/2611.17563#S4.T1)\.
Table 10:Gemma\-2 9B, per\-benchmark synthesis accuracy \(%\)\.Column names as in Table[1](https://arxiv.org/html/2611.17563#S4.T1)\. For the trained arms each cell is the mean±\\pmsd over three training seeds, each the mean of eight sampled answers; for Base and Self\-Refl, the sd is over the eight sampled answers per question, and Maj@64 votes over 64 samples per question, with the sd over three such draws\. Errored rows are excluded from every denominator; their rate is 0\.00 %–0\.23 % on every cell\.*AIME 2025 is at most 0\.3 for every arm*, so it lowers every macro by roughly a tenth and contributes almost nothing to the spread\.One row carries no reading\. AIME 2025 is at most 0\.3 for every arm, too close to zero to rank the arms: with thirty problems a single correct answer moves a cell by a few tenths, so no cell is emphasized and no claim rests on the row\.
Otherwise Self\-Disagreement leads on every benchmark — all but AIME 2025, just described, and Omni\-MATH, where SCoRe is ahead of it by 1\.4 points — and the macro average\.
## Appendix MReward\-hacking controls
The reward pays for correcting a flawed first answer, which creates an obvious exploit: degrade the first answer so that there is always something to correct\. The exploit is real and has been observed in this codebase, so we treat it as a check that must pass rather than as an objection to be argued away\.
The first answer is not sacrificed:Compared at matched steps, first\-answer accuracy does not fall in the dialectical arms\.*This is a control and not a result:*training raises it in every arm including the unrewarded\-dialectic one, so it does not distinguish the treatment and no claim in this paper rests on it\.
The synthesis is not copied from the challenge:The sharper check, because it tests the route by which a gain could be manufactured\. The rate at which the final answer simply reproduces the challenger’s falls from0\.6930\.693in the unrewarded\-dialectic arm to0\.3930\.393–0\.4190\.419in the arms that pay for the verdict\.*The arms that defend better are the arms that copy less*, so the gain cannot be attributed to adopting the challenge more often\. The arm with the verdict term removed sits at0\.7050\.705, with the control rather than the treatment, which is the internal check that the effect follows the variable we manipulated\.
No format exploitation:Pooled over each training run, format\-failure rates are below22% on both the first and the final answer in every arm, except SCoRe’s first attempt on Gemma\-2 9B \(4\.8–5\.4 %\), so the route in which a reward is captured by malformed output is not active\.
What these checks do not establish:They are a conjunction, and they are one\-sided\. The opposite failure — refusing a challenge that deserved to be accepted — passes every check above, which is why it is priced directly in the objective and measured separately rather than left to these controls\.
## Appendix NSCoRe, implemented as a trained baseline
SCoRe\([Kumar et al\., 2024](https://arxiv.org/html/2611.17563#bib.bib21)\)trains a single model to correct its own answer with reinforcement learning: the model answers, is told that its solution may contain an error, and answers again, with no opposing argument and no verdict\. It is the published recipe closest to Revise\-RL, which also revises without opposition, so we implemented it on our setup and trained it on both base models\. Tables[1](https://arxiv.org/html/2611.17563#S4.T1)–[3](https://arxiv.org/html/2611.17563#S4.T3)and[8](https://arxiv.org/html/2611.17563#A9.T8)–[14](https://arxiv.org/html/2611.17563#A17.T14)carry it beside the other arms, read with exactly their protocol\.
The recipe as we ran it:Each training trajectory has two turns\. The first is our thesis prompt, the question with its answer format, as for every arm\. The second appends SCoRe’s instruction to look again \(*“There might be an error in the solution above because of lack of understanding of the question\. Please correct the error, if any, and rewrite the solution\.”*\), with only its answer\-format clause replaced by ours\. Training has two stages, and each attempt earns11if its final answer is correct and00otherwise\. In Stage I only the second attempt is rewarded, and a KL penalty of0\.10\.1holds the first attempt near the base model\. In Stage II both attempts are rewarded, and the second also earns a progress bonusα\(r2−r1\)\\alpha\\,\(r\_\{2\}\-r\_\{1\}\)withα=10\\alpha=10\. A KL penalty of0\.010\.01applies to every turn in both stages, and, as for every arm, it enters the reward rather than the loss\. Rewards are per turn, with no discounting across turns\. Each sample’s advantage is taken against the other seven samples of its prompt \(RLOO\), inside the clipped update of §[B](https://arxiv.org/html/2611.17563#A2)\. Data, initialization, verifier, learning rate, batch, group size and training decoding are those of every arm on the same base model\. Stage II starts from Stage I’s weights and position in the data, with a fresh optimizer state\. Qwen3\-8B trains for250250steps,125125in each stage, and Gemma\-2 9B for100100,5050in each; each seed is read at its last step, the common step of its base model\.
## Appendix OThesis accuracy, benchmark by benchmark
Table[11](https://arxiv.org/html/2611.17563#A15.T11)gives each arm’s thesis accuracy, its first answer scored before any challenge, for the arms and training seeds of Tables[1](https://arxiv.org/html/2611.17563#S4.T1)and[10](https://arxiv.org/html/2611.17563#A12.T10)\. It is a control: a gain under challenge paid for out of the first answer would show here as a lower thesis accuracy, which the table does not show, since Self\-Disagreement’s thesis accuracy averages 55\.6 % on Qwen3\-8B and 36\.9 % on Gemma\-2 9B, against 54\.1–55\.3 and 32\.9–37\.1 for the other trained arms and 53\.3 and 27\.0 for the base model\.
Table 11:Thesis accuracy, per benchmark \(%\)\.The first answer, scored before any challenge, for the arms and training seeds of Tables[1](https://arxiv.org/html/2611.17563#S4.T1)and[10](https://arxiv.org/html/2611.17563#A12.T10); the column heads are Table[1](https://arxiv.org/html/2611.17563#S4.T1)’s\. Mean±\\pmsd over three training seeds, each the mean of eight sampled answers \(for Base, over its eight sampled answers\)\. An unusable answer scores zero\.
## Appendix PRepeated challenge
Every result in the main text comes from a single round of challenge: a first answer, one antithesis and one synthesis\. A model that holds once could still give way when the challenge is repeated, so this appendix asks whether the gain survives further rounds\. After three rounds, Self\-Disagreement still has the highest synthesis accuracy on both base models, and it is right in all three rounds far more often than any other arm \(Table[12](https://arxiv.org/html/2611.17563#A16.T12)\)\.
Setup:Round 1 is the dialectic on a D\-probe item: the model answers, writes its own antithesis and then a synthesis\. In each later round the same model writes a new antithesis against its current answer, in a fresh call that carries no history, and then a new synthesis that continues the conversation, so the model sees its first answer and every earlier challenge and synthesis\. We ran three rounds on the 327 D\-probe items, sampled four times per item, with each trained arm at its three training seeds\. A round that ends without a readable answer counts as wrong\.
Result:After round 3, Self\-Disagreement’s synthesis accuracy is 72\.0 % on Qwen3\-8B and 52\.4 % on Gemma\-2 9B, against at most 64\.8 % and 41\.7 % for every other arm, and it is right in all three rounds on 70\.0 % and 48\.1 % of items, against at most 53\.9 % and 26\.0 %, a lead that stays above 13 points in every 95 % interval over items\. It leads on both measures also when each round instead starts afresh, with the previous synthesis as the thesis\.
Table 12:Three rounds of challenge on the D\-probe \(%\)\.Synthesis accuracy after each round, and the share of items right in all three rounds, with the synthesis continuing one conversation\. Trained arms are the mean over three training seeds; bold is the best arm for each base model\.
## Appendix QThe same challenge for every arm
Every arm in the upper block of Table[1](https://arxiv.org/html/2611.17563#S4.T1)meets its own antithesis, so its synthesis accuracy reflects both how it decides and what it is asked to decide on\. Holding the challenge fixed separates the two: every arm receives the same first answer and the same antithesis from an outside model, and only the decision is its own\. The lead survives on both base models and against both challengers \(Table[1](https://arxiv.org/html/2611.17563#S4.T1), lower block\), and Table[13](https://arxiv.org/html/2611.17563#A17.T13)shows why: the arms trained on other objectives, and the base model given more inference, answer an antithesis much as the base model does, taking most of them whether right or wrong, while Self\-Disagreement holds against the wrong ones and still takes most of the right ones, and the base model made to switch as rarely does not catch up, so the advantage lies in which items it holds on, not in how many\.
Setup:For every item of the panel \(38,434 per base model, GPQA on its 250 items\), the base model’s first answer from its first run is challenged once by Gemini\-3\.8\-flash and once by Claude\-Haiku\-4\.5, each writing one antithesis with the challenging prompt of App\.[C](https://arxiv.org/html/2611.17563#A3), the prompt the base model itself answers\. Every arm writes the synthesis to that first answer and that antithesis with the harness, template and decoding of Table[1](https://arxiv.org/html/2611.17563#S4.T1), each trained arm at its three training seeds and the base model, each with eight sampled syntheses per item; Self\-Refl reviews each of those eight syntheses once, and Maj@64 is drawn three separate times, each draw the majority answer of 64 sampled syntheses\. Only the synthesis is sampled\. The outside antithesis is right on 36\.2 and 46\.1 % of the items against Qwen3\-8B’s first answers and on 40\.5 and 33\.1 % against Gemma\-2 9B’s \(Gemini\-3\.8\-flash and Claude\-Haiku\-4\.5, in that order; mean over the benchmarks\)\.
Deciding, not yielding less:On the items where exactly one of the first answer and the antithesis is right, the right decision is known, and an arm that only yields less trades one kind of error for the other\. Self\-Disagreement does more than that: it holds where holding is right on 45 to 78 % of these items, against 5 to 31 % for every other arm, while still yielding where yielding is right on 70 to 80 %, against 84 to 99 %, so the mean of the two rates puts it 8 to 19 points above every other arm, and the AUC of its switching \(the probability that an item where the antithesis is right draws more switches than one where it is wrong, ties counting half\) is the highest in every cell \(0\.68 to 0\.88, against at most 0\.81 for any other arm\)\.
More inference sharpens what the base model already does:Taking the majority of 64 samples turns the base model’s lean toward the antithesis into its answer, so it holds even less where holding is right \(5 to 22 %\) and yields even more where yielding is right \(94 to 99 %\), leaving its mean of the two rates within 3\.5 points of the base model’s\. A single review of each synthesis leaves Qwen3\-8B where it was and costs Gemma\-2 9B 4 to 5 points\. Neither supplies what training on the challenge does, which is the conclusion the upper block of Table[1](https://arxiv.org/html/2611.17563#S4.T1)reaches with each arm’s own antithesis\.
Switching less is not what makes it better:Self\-Disagreement switches on fewer of these items than the base model, but making the base model switch as rarely does not make it decide as well\. Pushed toward holding at its verdict word \(biases fixed in advance, up to 16 in log\-odds\), the base model changes its verdict and not its answer, and never comes down to Self\-Disagreement’s switch rate; pushed at its final answer, it does come down to it on Gemma\-2 9B against both challengers, and there Self\-Disagreement’s mean of the two rates is still higher by 10\.4 and 7\.0 points \(95 % intervals\[9\.1,11\.9\]\[9\.1,11\.9\]and\[4\.7,9\.5\]\[4\.7,9\.5\]\), and its synthesis accuracy by 7\.5 and 4\.4\. On Qwen3\-8B the pushes fixed in advance do not bring the base model down to that rate; larger ones added afterwards \(32 and 64, read from the same samples\) do, against both challengers, and there Self\-Disagreement’s mean of the two rates is higher by 17\.7 and 9\.9 points \(\[15\.3,18\.7\]\[15\.3,18\.7\]and\[7\.6,11\.9\]\[7\.6,11\.9\]\) and its synthesis accuracy by 12\.4 and 8\.3\. On both base models, then, its advantage lies in which items it holds on, not in how many\.
The disagreement probe, with the base model’s exchange for every arm:The same design on the probe of §[4](https://arxiv.org/html/2611.17563#S4): every arm receives the base model’s first answer to each of its 327 questions and the antithesis each of the seventeen challengers of Table[8](https://arxiv.org/html/2611.17563#A9.T8)wrote against it, and writes eight syntheses per training run \(Table[14](https://arxiv.org/html/2611.17563#A17.T14); 95 % intervals resample the questions, with the eight draws fixed\)\. With the exchange fixed, the odds ratios of §[4\.3](https://arxiv.org/html/2611.17563#S4.SS3)can be compared across models\. Self\-Disagreement’s log odds ratio is above the base model’s by 0\.68\[0\.40,0\.98\]\[0\.40,0\.98\]on Qwen3\-8B and 1\.23\[0\.89,1\.56\]\[0\.89,1\.56\]on Gemma\-2 9B, and above each of Verify\-RL, Revise\-RL, Accuracy\-RL and SCoRe by 0\.47 to 0\.66 and 0\.88 to 1\.12, every interval above zero; its mean of holding and yielding is above the base model’s by 5\.1\[2\.4,7\.7\]\[2\.4,7\.7\]and 17\.2\[14\.8,19\.6\]\[14\.8,19\.6\]points\. Paying for the verdict moves the balance toward holding: against Judge, which must give the verdict and is not paid for it, Self\-Disagreement holds 6\.2 points more\[5\.3,7\.1\]\[5\.3,7\.1\]and yields 7\.6 points less\[6\.3,9\.0\]\[6\.3,9\.0\], with no change in their mean \(−0\.7\-0\.7,\[−1\.5,\+0\.1\]\[\-1\.5,\+0\.1\]\) and a log odds ratio higher by 0\.20\[0\.11,0\.30\]\[0\.11,0\.30\]\.
Told to hold, the base model still decides worse:We also instructed the base model to keep its first answer, at three strengths fixed in advance, each replacing the template’s sentence “Update if the opposition is genuinely right; defend if it is wrong” with one of “Update only if you are confident the opposition is right; otherwise keep your answer”, “Your first answer is usually right\. Keep it unless the opposition shows a specific error in it that you can verify” and “Keep your first answer\. Change it only if the opposition proves beyond doubt that it is wrong”, on the probe with the base model’s exchange \(Table[14](https://arxiv.org/html/2611.17563#A17.T14)\)\. The prompted base model still yields on 81\.1 to 83\.7 % of the items where yielding is right on Qwen3\-8B and 92\.2 to 94\.4 % on Gemma\-2 9B, against Self\-Disagreement’s 58\.3 and 63\.0 %, and at every strength Self\-Disagreement’s mean of holding and yielding is higher, by 3\.5 to 5\.2 and 14\.4 to 19\.0 points, every interval above zero\.
Table 13:The same challenge, every arm: what each decides \(%\)\.Holds where holding is right, yields where yielding is right, their mean, and the AUC of switching, on the items where exactly one of the first answer and the antithesis is right; macro over benchmarks\.*Self\-Disagreement holds against a wrong antithesis far more often than any other arm while still taking most of the right ones, and it leads every arm on the mean and on the AUC\.*Table 14:The disagreement probe, with the base model’s exchange for every arm\.Every arm receives the base model’s first answer to each probe question and the antithesis each of the seventeen challengers of Table[8](https://arxiv.org/html/2611.17563#A9.T8)wrote against it\. Holds, yields and their mean \(%\) as in Table[13](https://arxiv.org/html/2611.17563#A17.T13); the log odds ratio of §[4\.3](https://arxiv.org/html/2611.17563#S4.SS3), averaged over training runs; the AUC of switching\. Three training runs per trained arm \(SelfD\-RL withλstub=2\\lambda\_\{\\mathrm\{stub\}\}=2: two\), eight draws each; Judge must give the verdict and is not paid for it\. The prompted rows are the base model told to keep its first answer, at three strengths \(App\.[Q](https://arxiv.org/html/2611.17563#A17)\), given by holding, yielding and their mean\.*Given the same exchange, Self\-Disagreement has the highest log odds ratio and AUC on both base models\.*
## Appendix RPBT’s released model in our setting
PBT\([Stengel\-Eskin et al\., 2025](https://arxiv.org/html/2611.17563#bib.bib40)\)trains a model to accept persuasion toward a right answer and to resist it toward a wrong one, the two\-sided target we share \(App\.[A](https://arxiv.org/html/2611.17563#A1)\)\. We measure its released model in our setting, beside the model it starts from, Llama\-3\.1\-8B \(instruction\-tuned\), and SelfD\-RL trained from the same model \(one training run, step 150\), with the paper’s grader \(Table[15](https://arxiv.org/html/2611.17563#A18.T15)\)\.
Against its own antithesis:On the disagreement probe, PBT’s synthesis accuracy and capitulation are close to the base model’s \(51\.4 and 4\.2 % against 53\.8 and 4\.4 %\), and SelfD\-RL is ahead on both \(58\.5 and 2\.2 %\)\. Table[4](https://arxiv.org/html/2611.17563#S4.T4)reads the same base model on validation during training, over two seeds, hence its different values\.
Given the same exchange:With the base model’s first answer and an outside challenger’s antithesis \(Gemini\-3\.8\-flash and Claude\-Haiku\-4\.5\), PBT moves the base model toward holding, 15\.1 points more holding and 14\.7 points less yielding, with their mean level with the base model’s \(\+0\.2\+0\.2points,\[−2\.6,\+2\.9\]\[\-2\.6,\+2\.9\]\)\. SelfD\-RL leads PBT on the mean by 5\.7 points\[2\.7,8\.6\]\[2\.7,8\.6\]and on the AUC by 0\.140\[0\.085,0\.200\]\[0\.085,0\.200\], and the base model by 5\.9 points\[3\.5,8\.4\]\[3\.5,8\.4\]and 0\.114\[0\.060,0\.166\]\[0\.060,0\.166\]; 95 % intervals resample the questions, with the draws fixed\. The models were trained on different data, so this compares released models, not methods\.
Table 15:PBT’s released model in our setting, Llama\-3\.1\-8B\.\(a\) Each model against its own antithesis on the disagreement probe, mean±\\pmsd over four draws\. \(b\) Each model given the base model’s first answer and an outside antithesis \(Gemini\-3\.8\-flash and Claude\-Haiku\-4\.5\), eight draws: holds, yields and their mean \(%\) over both challengers, the AUC of switching averaged over them, and synthesis accuracy\. SelfD\-RL: one training run, step 150\.Similar Articles
Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR
This paper introduces RLRT, a method that reverses teacher signals in self-distillation to reinforce successful student deviations, enhancing reasoning exploration in large language models.
@sebkrier: Training models with RL can often lead to the reward signal being gamed; for example when you use an LLM judge for fuzz…
Debate training can mitigate reward hacking in reinforcement learning from AI feedback (RLAIF) by using a debate opponent to improve ground-truth accuracy for fuzzy tasks.
ICRL: Learning to Internalize Self-Critique with Reinforcement Learning
This paper introduces ICRL, a framework that jointly trains a solver and critic with reinforcement learning to internalize critique guidance, enabling the solver to improve without external critique. It uses distribution calibration and role-wise group advantage estimation, achieving 6-7 point gains over GRPO on agentic and mathematical reasoning tasks.
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
This paper introduces Negative Self-Distillation (NSD), a framework for improving large language model reasoning by diverging from flawed reasoning instead of imitating privileged solutions, showing consistent gains over existing methods on mathematical benchmarks.
Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging
This paper introduces EYT-Bench, a human-centered benchmark for evaluating LLMs in multi-turn dialogues with a decoupled user simulation, target modeling, and judging design. It reveals that closed- and open-source models differ significantly on objective intent-tracking but are similar on subjective dimensions, and that reasoning improves objective tracking while persona format strongly affects trajectory spread.