RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization

# RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization
Source: [https://arxiv.org/html/2605.04539](https://arxiv.org/html/2605.04539)
Qiming Bao University of Auckland qbao775@aucklanduni\.ac\.nz&Juho Leinonen Aalto University juho\.2\.leinonen@aalto\.fi&Paul Denny University of Auckland p\.denny@auckland\.ac\.nz&Michael J\. Witbrock University of Auckland m\.witbrock@auckland\.ac\.nz

###### Abstract

Direct Preference Optimization \(DPO\) has emerged as an efficient alternative to PPO\-based RLHF, yet its application to knowledge\-intensive generation tasks reveals a fundamental limitation: standard preference signals—whether from human annotators or LLM judges—exhibit a systematicverbosity biasthat rewards linguistic fluency over logical correctness\. We show empirically that this reward signal blindspot leaves a large logical alignment gap: SFT\-trained models achieve NLI entailment scores of only 0\.05–0\.22, despite producing fluent, confident\-sounding text\. We proposeRLearner\-LLM, a framework that resolves this throughHybrid\-DPO: an automated preference construction pipeline that fuses a DeBERTa\-v3 NLI entailment signal with a verifier LLM score, eliminating the need for human annotation while overcoming the “alignment tax” that degrades fluency under single\-signal optimization\. Evaluated across five academic domains \(Biology, Medicine, Law\) with three base architectures \(LLaMA\-2\-13B, Qwen3\-8B, and Gemma 4 E4B\-it\), RLearner\-LLM achieves up to 6×\\timesNLI improvement over SFT baselines, with NLI gains in1111of1515\(architecture, domain\) cells and consistent answer\-coverage gains\. On Gemma 4 E4B\-it—a 4\.5B\-effective\-parameter dense model—Hybrid\-DPO lifts NLI entailment in four of five domains \(from \+11\.9% on Auckland Law to \+2\.4×\\timeson UK Medicine Year 2\), with faster inference across all five, demonstrating that the approach scales down to compact modern base models without losing the alignment\-tax mitigation\. We further show these gains are robust to aggressive, response\-grounded question\-quality filtering: under astrict\-discriminatorfilter that keeps only questions whose author key matches the modal student answer \(computed from per\-option student response logs\), answer\-coverage improves in every architecture–corpus cell and NLI in four of six\. Our Qwen3\-8B RLearner\-LLM wins95%95\\%of blind pairwise comparisons against its own SFT baseline, while GPT\-4o\-mini in turn wins95%95\\%against our concise RLearner\-LLM output—a result that, viewed alongside the69%69\\%win rate the same judge awards a verbose SFT baseline over our DPO\-aligned model, reproduces the verbosity bias we document on a frontier comparator and reinforces the case for logic\-aware automatic metrics \(NLI, ACR\) over LLM\-as\-a\-judge on knowledge\-intensive generation\.111Code, training scripts, and per\-domain evaluation artefacts:[https://github\.com/14H034160212/Explanation\-Generation](https://github.com/14H034160212/Explanation-Generation)

## 1Introduction

The generation of high\-quality educational explanations represents a fundamental challenge in natural language processing\. Unlike general\-purpose conversational tasks, educational rationales must simultaneously satisfy two competing requirements: they must belogically grounded—deriving the correct answer through sound deductive steps—andlinguistically fluent—maintaining natural, readable prose appropriate for a learner audience\.

Supervised Fine\-Tuning \(SFT\) on teacher\-generated demonstrations has become the standard approach to instilling this capability into LLMs\. However, SFT models exhibit a consistent and critical failure mode: while they produce fluent, confident\-sounding text, they rarely construct the deductive chain that justifieswhythe correct answer follows from the question context\. Empirically, SFT models in our study achieve NLI entailment scores of only 0\.05–0\.22, indicating that the generated explanation rarely logically implies the ground\-truth answer—even when the model “knows” the fact\.

Reinforcement Learning from Human Feedback \(RLHF\) and its efficient derivative, Direct Preference Optimization \(DPO\)[rafailov2023direct](https://arxiv.org/html/2605.04539#bib.bib3), have sought to move beyond SFT by aligning models to human preferences\. Yet these approaches contain a structural flaw when applied to factually demanding tasks:the preference signal does not measure logical correctness\. Human annotators—and LLM judges—exhibit a well\-documentedverbosity bias, systematically preferring longer, rhetorically polished responses over concise, logically precise ones[zheng2024judging](https://arxiv.org/html/2605.04539#bib.bib10)\. As we demonstrate empirically, GPT\-4o\-mini awards a 69% win rate to a verbose SFT baseline over a more logically precise but concise RL\-aligned model\. DPO trained on such signals therefore reinforces stylistic fluency at the expense of factual grounding, leaving the core alignment gap unaddressed\.

A natural remedy is to replace human preference labels with an automated logical metric—specifically, NLI entailment probability\. However, optimizing DPO solely against NLI entailment incurs what we term thealignment tax: the model learns to produce short, repetitive snippets that satisfy the NLI scorer but collapse in linguistic coherence\. This trade\-off between logical soundness and readability represents a fundamental tension that single\-dimensional reward signals cannot resolve\.

Linguistic Fluency / QualityLogical Entailment \(NLI\)StandardAlignment FrontierPure Logic \(NLI Opt\)Pure Fluency \(SFT/Style Opt\)RLearner\-LLM \(Ours\)Alignment TaxFigure 1:Conceptual illustration of theAlignment Tax\. Standard optimization forces a trade\-off between logical grounding and linguistic fluency\. RLearner\-LLM pushes the Pareto frontier toward the upper\-right quadrant through a dual\-signal hybrid reward\.In this paper, we proposeRLearner\-LLM, a framework that resolves this tension by addressing the root cause directly: the reward signal used to construct preference pairs\. Our key insight is that logical grounding and linguistic fluency arecomplementary objectives, not competing ones—and that an automated dual\-signal reward combining NLI entailment \(for logical soundness\) with a verifier score \(for pedagogical quality\) can construct high\-fidelity preference pairs without human annotation, while simultaneously avoiding the degenerate solutions that plague single\-signal optimization\.

Our contributions are as follows:

1. 1\.Problem diagnosis: We identify that the root cause of logical misalignment in DPO\-trained educational models is not the optimization algorithm itself, but thereward signal blindspot—the inability of standard preference signals to measure logical correctness\. We provide empirical evidence of verbosity bias in LLM\-as\-a\-judge evaluation that perpetuates this blindspot\.
2. 2\.Hybrid\-DPO framework: We introduceRLearner\-LLM, which automatically constructs preference pairs𝒟p​r​e​f\\mathcal\{D\}\_\{pref\}using a composite scoreH​\(Ei\)=0\.5⋅SNLI​\(Ei\)\+0\.5⋅S~Verifier​\(Ei\)H\(E\_\{i\}\)=0\.5\\cdot S\_\{\\text\{NLI\}\}\(E\_\{i\}\)\+0\.5\\cdot\\tilde\{S\}\_\{\\text\{Verifier\}\}\(E\_\{i\}\), fusing a DeBERTa\-v3 NLI signal with a min\-max\-normalised verifier LLM score \(Eq\.[2](https://arxiv.org/html/2605.04539#S3.E2)\)\. The dual\-signal hybrid eliminates the need for human annotation while overcoming the alignment tax\.
3. 3\.Cross\-architecture validation: We validate across three base architectures \(LLaMA\-2\-13B, Qwen3\-8B, and Gemma 4 E4B\-it\) and five academic domains \(Biology, Medicine, Law\), demonstrating up to 6×\\timesNLI improvement over SFT baselines in1111of1515\(architecture, domain\) cells, with consistent ACR gains on stronger base models\. Gemma 4 E4B\-it—a 4\.5B\-effective\-parameter dense model—shows Hybrid\-DPO NLI gains in four of five domains \(\+11\.9% Auckland Law, \+32\.0% UK Medicine Y1, \+65\.6% Cardiff Biology, \+2\.4×\\timesUK Medicine Y2\), with faster inference across all five and the first single\-pass\-RL result to surpass the iterative ILearner\-LLM \(K=5\) baseline on Auckland Law NLI\. Our Qwen3\-8B model further wins95%95\\%of blind pairwise comparisons against its own SFT baseline; on the same judge it loses95%95\\%to GPT\-4o\-mini’s longer outputs, replicating the verbosity bias on a frontier comparator and motivating logic\-aware automatic metrics over LLM\-as\-a\-judge for knowledge\-intensive generation\.
4. 4\.Robustness to question\-quality filtering: Using the full PeerWise answer\-submission logs, we show the gains survive astrict\-discriminatorfilter that keeps only questions whose author key matches the*modal student answer*: across three architectures and two corpora, answer\-coverage improves in all six \(architecture, corpus\) cells and NLI in four of six, reproducing the per\-architecture pattern of the main results \(Appendix[G](https://arxiv.org/html/2605.04539#A7)\)\.

## 2Related Work

### 2\.1Alignment and Preference Optimization

Optimizing Large Language Models \(LLMs\) to align with human intent has predominantly relied on Reinforcement Learning from Human Feedback \(RLHF\)[ouyang2022training](https://arxiv.org/html/2605.04539#bib.bib1), utilizing Proximal Policy Optimization \(PPO\)[schulman2017proximal](https://arxiv.org/html/2605.04539#bib.bib2)\. Recently, Direct Preference Optimization \(DPO\)[rafailov2023direct](https://arxiv.org/html/2605.04539#bib.bib3)emerged as a stable alternative that mathematically bypasses the need for an explicit reward model\. Building on this, recent advancements like RRHF[yuan2023rrhf](https://arxiv.org/html/2605.04539#bib.bib6), ORPO[hong2024orpo](https://arxiv.org/html/2605.04539#bib.bib7), and SimPO[meng2024simpo](https://arxiv.org/html/2605.04539#bib.bib8)have further simplified alignment via reference\-free objectives\. The state\-of\-the\-art in 2025 has moved toward adaptive batch\-wise scheduling[zhang2025adaptive](https://arxiv.org/html/2605.04539#bib.bib18)and iterative dual\-based methods for constrained alignment[chen2025constrained](https://arxiv.org/html/2605.04539#bib.bib19), addressing the inherent algorithmic biases identified in standard RLHF[ji2024limits](https://arxiv.org/html/2605.04539#bib.bib23);[shrivastava2024algorithmic](https://arxiv.org/html/2605.04539#bib.bib24)\.

### 2\.2Hallucination Mitigation and Reasoning

In knowledge\-intensive tasks, such as educational explanation generation, factual correctness is paramount\. Prior work has explored grounding LLMs using retrieval\-augmented generation \(RAG\)[lewis2020retrieval](https://arxiv.org/html/2605.04539#bib.bib4)or process\-based feedback[uesato2023verify](https://arxiv.org/html/2605.04539#bib.bib5)\. Modern techniques focus on mechanistic hallucination detection[li2023haludetect](https://arxiv.org/html/2605.04539#bib.bib15)and Multi\-Model Contrastive Decoding \(MCD\)[liu2025contrastive](https://arxiv.org/html/2605.04539#bib.bib20)\. While debates continue regarding whether emergent abilities are a metric\-driven "mirage"[schaeffer2023mirage](https://arxiv.org/html/2605.04539#bib.bib14), advancements in 2025 demonstrate that reasoning quality can be significantly enhanced by verifying "decision pivots" in Chain\-of\-Thought traces[wang2025pivots](https://arxiv.org/html/2605.04539#bib.bib17)or equipping models with cognitive tools[zhao2025cognitive](https://arxiv.org/html/2605.04539#bib.bib21)\. However, these logical optimizations often degrade fluency—the "alignment tax"—which our work addresses via a hybrid verifier\-consistent reward\.

### 2\.3LLM\-as\-a\-Judge and Evaluation Bias

The use of powerful instruction\-tuned models like GPT\-4 as automated evaluators[zheng2024judging](https://arxiv.org/html/2605.04539#bib.bib10)is widespread but prone to significant biases, including verbosity bias[wang2023large](https://arxiv.org/html/2605.04539#bib.bib11)and self\-preference bias[pan2024evalbias](https://arxiv.org/html/2605.04539#bib.bib12)\. The verbosity\-bias confound has been formally analysed in DPO itself:[park2024lengthbias](https://arxiv.org/html/2605.04539#bib.bib25)show that standard DPO and itsβ\\beta\-tuned variants[park2024betadpo](https://arxiv.org/html/2605.04539#bib.bib9)systematically conflate explanation length with quality, and propose length\-aware regularisation as a partial remedy\. Our work targets the*upstream*reward\-construction stage rather than the downstream loss: by replacing length\-correlated stylistic preferences with an NLI\-anchored dual\-signal reward \(Section[3\.1](https://arxiv.org/html/2605.04539#S3.SS1)\) we cut the bias at its origin, which the head\-to\-head Table[3](https://arxiv.org/html/2605.04539#S5.T3)confirms persists on frontier comparators despite length controls applied at the output stage\. Comprehensive assessments like DecodingTrust[wang2023trust](https://arxiv.org/html/2605.04539#bib.bib13)further highlight the vulnerability of current models to stylistic manipulation; we maintain safety and knowledge\-retention standards[kim2025safety](https://arxiv.org/html/2605.04539#bib.bib22)throughout\.

Stage 1: Stochastic GenerationContext\(C,Q\)\(C,\\,Q\)Generatorπθ\\pi\_\{\\theta\}Explanationsℰ=\{Ei\}i=1n\\mathcal\{E\}=\\\{E\_\{i\}\\\}\_\{i=1\}^\{n\}nnsamplesStage 2: Logic\-Driven Reward & AlignmentNLI EntailmentSnli​\(Ei\)S\_\{\\text\{nli\}\}\(E\_\{i\}\)Stylistic VerifierSver​\(Ei\)S\_\{\\text\{ver\}\}\(E\_\{i\}\)Length Penaltyγ​ℓ​\(Ei\)\\gamma\\,\\ell\(E\_\{i\}\)Hybrid Reward\(additive A or multiplicative\-ACR M, selected per training pool\)\(A\)HA​\(Ei\)=0\.5​Snli​\(Ei\)\+0\.5​S~ver​\(Ei\)H\_\{A\}\(E\_\{i\}\)\\;=\\;0\.5\\,S\_\{\\text\{nli\}\}\(E\_\{i\}\)\\;\+\\;0\.5\\,\\tilde\{S\}\_\{\\text\{ver\}\}\(E\_\{i\}\)\(M\)HM​\(Ei\)=\(wnli​Snli⋅wver​S~ver−γ​ℓ\)⋅1​\[acr​\(Ei\)≥θ\]H\_\{M\}\(E\_\{i\}\)\\;=\\;\\bigl\(w\_\{\\text\{nli\}\}\\,S\_\{\\text\{nli\}\}\\,\\cdot\\,w\_\{\\text\{ver\}\}\\,\\tilde\{S\}\_\{\\text\{ver\}\}\\;\-\\;\\gamma\\,\\ell\\bigr\)\\;\\cdot\\;\\mathbf\{1\}\\\!\\bigl\[\\mathrm\{acr\}\(E\_\{i\}\)\\geq\\theta\\bigr\]reward signalHybrid Direct Preference Optimisation \(Hybrid\-DPO\)preference updateFigure 2:TheRLearner\-LLMframework\. Stage 1 samplesnncandidate explanations from the generatorπθ\\pi\_\{\\theta\}\. Stage 2 scores each candidate with the dual\-signalHybrid Reward, instantiated as either the*additive*variantHAH\_\{A\}or the*multiplicative\-ACR*variantHMH\_\{M\}\(§[3\.1](https://arxiv.org/html/2605.04539#S3.SS1); selector rule: pool≥∼150\\geq\{\\sim\}150pairs and a single/aligned domain→\\toM, otherwise→\\toA\)\. NLI entailment and the verifier score are used by both variants; the length\-penalty input and the ACR indicator𝟏​\[acr≥θ\]\\mathbf\{1\}\[\\mathrm\{acr\}\\geq\\theta\]enterHMH\_\{M\}only\. Preference pairs are then formed from the scored pool and used to updateπθ\\pi\_\{\\theta\}via Hybrid\-DPO\. The dual\-signal hypothesis \(NLI⊕\\oplusVerifier\) is invariant across both variants; only the algebraic combination differs\.S~ver=\(Sver−1\)/4\\tilde\{S\}\_\{\\text\{ver\}\}\{=\}\(S\_\{\\text\{ver\}\}\{\-\}1\)/4denotes the\[0,1\]\[0,1\]\-normalized verifier score\.

## 3Methodology

### 3\.1First\-Principles Derivation of the Hybrid Reward

#### The RLHF Objective and Its Core Tension\.

The canonical RLHF objective[ouyang2022training](https://arxiv.org/html/2605.04539#bib.bib1)is:

maxπθ⁡𝔼y∼πθ​\(y\|x\)​\[r​\(x,y\)\]−β​KL​\(πθ∥πref\)\\max\_\{\\pi\_\{\\theta\}\}\\;\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\(y\|x\)\}\\bigl\[r\(x,y\)\\bigr\]\-\\beta\\,\\text\{KL\}\\bigl\(\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\text\{ref\}\}\\bigr\)\(1\)whereπref\\pi\_\{\\text\{ref\}\}is the supervised fine\-tuned reference policy andβ\\betacontrols how much the learned policy is allowed to deviate from it\. DPO[rafailov2023direct](https://arxiv.org/html/2605.04539#bib.bib3)derives the closed\-form optimal policyπ∗​\(y\|x\)∝πref​\(y\|x\)⋅exp⁡\(r​\(x,y\)/β\)\\pi^\{\*\}\(y\|x\)\\propto\\pi\_\{\\text\{ref\}\}\(y\|x\)\\cdot\\exp\(r\(x,y\)/\\beta\)and reformulates the objective as a direct binary cross\-entropy loss on preference pairs, eliminating the need for an explicit reward model\.

Thealignment taxarises from a*gradient conflict*: the KL term in Eq\.[1](https://arxiv.org/html/2605.04539#S3.E1)regularisesπθ\\pi\_\{\\theta\}toward an MLE\-trainedπref\\pi\_\{\\text\{ref\}\}that prefers verbose, surface\-coherent text, while a logic\-focused reward \(e\.g\. pure NLI entailment\) rewards short dense deductive chains off that fluency manifold\. Single\-signal optimisation under this conflict either collapses into degenerate snippets that satisfy NLI at the cost of coherence, or capitulates to the KL prior and produces fluent\-but\-logically\-weak text—both manifest as alignment tax on the frontier in Fig\.[1](https://arxiv.org/html/2605.04539#S1.F1)\.

#### The Dual\-Signal Hybrid Reward\.

The load\-bearing design choice is to score candidates with both an NLI entailment scoreSNLI​\(E\)∈\[0,1\]S\_\{\\text\{NLI\}\}\(E\)\\in\[0,1\]and a min\-max\-normalised verifier scoreS~ver​\(E\)=\(Sver​\(E\)−1\)/4∈\[0,1\]\\tilde\{S\}\_\{\\text\{ver\}\}\(E\)\{=\}\(S\_\{\\text\{ver\}\}\(E\)\{\-\}1\)/4\\in\[0,1\]\. Used alone each signal degenerates \(pure NLI yields answer\-repeating snippets; pure verifier reintroduces verbosity bias[wang2023large](https://arxiv.org/html/2605.04539#bib.bib11);[park2024lengthbias](https://arxiv.org/html/2605.04539#bib.bib25)\)\. We instantiate the Hybrid reward at two operating points:

HA​\(E\)\\displaystyle H\_\{A\}\(E\)=0\.5​SNLI\+0\.5​S~ver,\\displaystyle=0\.5\\,S\_\{\\text\{NLI\}\}\+0\.5\\,\\tilde\{S\}\_\{\\text\{ver\}\},\(2a\)HM​\(E\)\\displaystyle H\_\{M\}\(E\)=\(wnli​SNLI⋅wver​S~ver−γ​ℓnorm\)⋅𝟏​\[ACR≥θ\]\.\\displaystyle=\\bigl\(w\_\{\\text\{nli\}\}\\,S\_\{\\text\{NLI\}\}\\cdot w\_\{\\text\{ver\}\}\\,\\tilde\{S\}\_\{\\text\{ver\}\}\-\\gamma\\,\\ell\_\{\\text\{norm\}\}\\bigr\)\\cdot\\mathbf\{1\}\[\\mathrm\{ACR\}\{\\geq\}\\theta\]\.\(2b\)The additiveHAH\_\{A\}\(Eq\.[2a](https://arxiv.org/html/2605.04539#S3.E2.1)\) admits chosen pairs when*either*signal is sufficiently better than the rejected counterpart \(high recall\)\. The multiplicative\-ACRHMH\_\{M\}\(Eq\.[2b](https://arxiv.org/html/2605.04539#S3.E2.2)\) requires*both*signals to favour the chosen candidate via the product, hard\-gates onACR​\(E\)≥θ\\mathrm\{ACR\}\(E\)\{\\geq\}\\thetato drop answer\-evasive candidates, and subtracts a length penaltyγ​ℓnorm​\(E\)\\gamma\\,\\ell\_\{\\text\{norm\}\}\(E\)to suppress the length confounder \(high precision\)\.HAH\_\{A\}contains no length term: in cross\-domain pools the heterogeneous length distribution destabilises a singleγ\\gamma, andHAH\_\{A\}is preferred precisely there\.

#### Selector rule\.

HMH\_\{M\}retains∼2\\sim\\\!2–4×4\\timesfewer preference pairs thanHAH\_\{A\}on the same candidate pool \(the hard gate drops∼40%\{\\sim\}40\\%of candidates, and the product collapses many surviving pairs below the score\-gap threshold\)\. We default toHMH\_\{M\}and fall back toHAH\_\{A\}when the projectedHMH\_\{M\}pool drops below∼150\{\\sim\}150pairs or the training corpus merges domains with disparate answer\-keyword distributions\. This rule is post\-hoc rationalisation of our experimental progression; the head\-to\-head Table[4](https://arxiv.org/html/2605.04539#S5.T4)showsHAH\_\{A\}andHMH\_\{M\}within∼1\{\\sim\}1pp NLI on average across1111matched cells,HMH\_\{M\}winning7/117/11\. Both variants beat NLI\-only \(0\.14190\.1419\) and verifier\-only \(0\.13680\.1368\) DPO on Qwen3\-8B Cardiff \(SFT0\.19590\.1959\); the dual\-signal hypothesis, not the algebraic form, is the load\-bearing claim\.

### 3\.2Preference Pair Construction

Givennncandidate explanations\{Ei\}i=1n\\\{E\_\{i\}\\\}\_\{i=1\}^\{n\}sampled fromπθ\\pi\_\{\\theta\}for\(C,Q\)\(C,Q\), we score each withH​\(Ei\)H\(E\_\{i\}\)and retain pairs\(Ei,Ej\)\(E\_\{i\},E\_\{j\}\)withH​\(Ei\)\>H​\(Ej\)\+ΔH\(E\_\{i\}\)\>H\(E\_\{j\}\)\+\\Deltaas \(chosen, rejected\), whereΔ≥0\.05\\Delta\{\\geq\}0\.05\. UnderHMH\_\{M\}, the ACR gate is a hard pre\-filter: only pairs whose chosen member passesACR≥θ\\mathrm\{ACR\}\{\\geq\}\\thetaare eligible, ruling out the rank\-inversion edge case where a long but valid candidate could receiveHM​\(E\)<0H\_\{M\}\(E\)\{<\}0from the length penalty \(we verified empirically that this corner case produces no inversions in our 5\-domain pool\)\.

### 3\.3Hybrid\-DPO Training Objective

The Hybrid\-DPO objective follows the standard DPO formulation[rafailov2023direct](https://arxiv.org/html/2605.04539#bib.bib3)but is trained on𝒟pref\\mathcal\{D\}\_\{\\text\{pref\}\}constructed with the Hybrid reward \(Eq\.[2](https://arxiv.org/html/2605.04539#S3.E2)\):

ℒDPO​\(πθ\)=−𝔼\(x,y\+,y−\)∼𝒟pref​\[log⁡σ​\(β​\(log⁡πθ​\(y\+\|x\)πref​\(y\+\|x\)−log⁡πθ​\(y−\|x\)πref​\(y−\|x\)\)\)\]\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\_\{\\theta\}\)=\-\\mathbb\{E\}\_\{\(x,y^\{\+\},y^\{\-\}\)\\sim\\mathcal\{D\}\_\{\\text\{pref\}\}\}\\\!\\left\[\\log\\sigma\\\!\\left\(\\beta\\\!\\left\(\\log\\frac\{\\pi\_\{\\theta\}\(y^\{\+\}\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y^\{\+\}\|x\)\}\-\\log\\frac\{\\pi\_\{\\theta\}\(y^\{\-\}\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y^\{\-\}\|x\)\}\\right\)\\right\)\\right\]\(3\)Because every chosen exampley\+y^\{\+\}in𝒟pref\\mathcal\{D\}\_\{\\text\{pref\}\}outscores its rejected counterpart on*both*dual\-signal axes—NLI entailment and verifier fluency—regardless of which Hybrid variant \(HAH\_\{A\}orHMH\_\{M\}\) constructed the pair, the gradient ofℒDPO\\mathcal\{L\}\_\{\\text\{DPO\}\}consistently points toward higher logical entailment without sacrificing pedagogical quality\. This is the mechanism by which the Hybrid reward breaks the fluency attractor that causes alignment tax in single\-signal optimisation\.

#### Implementation notes\.

Two base models in this study require non\-default LoRA target selection\. Qwen3\-8B applies RMSNorm toq,kq,kafter the projection reshape, so LoRA adapters onq\_proj/k\_projwould corrupt that path; we restrict its targets tov\_proj,o\_proj, and the three MLP projections\. Gemma 4 E4B\-it ships as a multimodalGemma4ForConditionalGenerationcheckpoint whose text\-tower weights live undermodel\.language\_model\.\*; we therefore load that class explicitly and use a full\-path LoRA target regex that matches exactly the258258plainnn\.Linearattention/MLP projections in the text tower \(full pattern in Appendix[C](https://arxiv.org/html/2605.04539#A3)\)\. At inference, the DPO adapter must be composed on top of the SFT\-merged base \(base→apply SFT→merge→apply DPO→merge\\text\{base\}\\to\\text\{apply SFT\}\\to\\text\{merge\}\\to\\text\{apply DPO\}\\to\\text\{merge\}\), since the DPO delta was trained relative to that reference point\.

## 4Experimental Setup

We conduct experiments on three base architectures spanning two parameter regimes and three pre\-training recipes:LLaMA\-2\-13B,Qwen3\-8B, andGemma 4 E4B\-it\(4\.54\.5B effective parameters via Per\-Layer Embeddings\)\. All training is performed on a cluster of NVIDIA A100 GPUs using the TRL and PEFT libraries\. The same iterative\-refinementILearner\-LLM \(K=5\)baseline of[bao2025exploring](https://arxiv.org/html/2605.04539#bib.bib16)is reproduced from a LLaMA\-2\-13B SFT checkpoint withK=5K\{=\}5refinement passes per question, identical to the original publication; we use this as a strictly stronger inference\-time baseline \(5×5\\timesthe inference compute of our single\-pass models\)\.

### 4\.1Datasets and Preprocessing

The evaluation spans five academic domains: Cardiff Biology, Sydney Biology, Auckland Law, and UK Medicine \(Years 1 and 2\)\. For each domain, we utilize a 100\-question held\-out test set\. Input contexts are formatted as multi\-choice questions with associated correct options\. The SFT corpus consists of 13,211 question–explanation pairs authored byundergraduate studentson the PeerWise peer\-learning platform\. These explanations reflect authentic learner reasoning rather than expert\-curated answer keys, which introduces a known ceiling on ground\-truth quality \(incomplete justifications, occasional misconceptions, and stylistic variance\); we quantify the downstream effect of this ceiling through the SFT failure\-mode analysis in Table[10](https://arxiv.org/html/2605.04539#A4.T10)and discuss its implications for future expert\-verified corpora in Section[6\.3](https://arxiv.org/html/2605.04539#S6.SS3)\.

### 4\.2Training Configurations

SFT: 3 epochs on13,21113\{,\}211examples, LoRAr=16r\{=\}16,α=32\\alpha\{=\}32, per\-device batch size22, gradient accumulation88, bf16\.Preference data:500500prompts×\\timesn=3n\{=\}3candidates scored withH​\(E\)H\(E\)\(Eq\.[2](https://arxiv.org/html/2605.04539#S3.E2)\)\.HMH\_\{M\}useswnli=0\.7w\_\{\\text\{nli\}\}\{=\}0\.7,wver=0\.3w\_\{\\text\{ver\}\}\{=\}0\.3,γ=0\.002×\(word\_count/100\)\\gamma\{=\}0\.002\\times\(\\text\{word\\\_count\}/100\),θ=0\.5\\theta\{=\}0\.5\(ACR\-failing candidates get zero reward\);HAH\_\{A\}uses0\.5/0\.50\.5/0\.5weights onSNLIS\_\{\\text\{NLI\}\}andS~ver=\(Sver−1\)/4\\tilde\{S\}\_\{\\text\{ver\}\}\{=\}\(S\_\{\\text\{ver\}\}\{\-\}1\)/4with no gate or length term\. Pairs with score gapΔ≥0\.05\\Delta\{\\geq\}0\.05are retained\.DPO: 5 epochs, lr5×10−55\\times 10^\{\-5\}, batch size11, grad accum88, bf16,βDPO=0\.1\\beta\_\{\\text\{DPO\}\}\{=\}0\.1\.Variant assignment\(selector rule, §[3\.1](https://arxiv.org/html/2605.04539#S3.SS1)\): LLaMA\-2\-13B usesHAH\_\{A\}on the cross\-domain pool \(239239pairs\); Qwen3\-8B usesHMH\_\{M\}atN=5N\{=\}5on Cardiff/Sydney \(143143pairs\) andHAH\_\{A\}cross\-domain elsewhere \(632632pairs\); Gemma 4 E4B\-it usesHMH\_\{M\}on the 5\-domain merged pool \(164164pairs\)\. Full assignment in Table[11](https://arxiv.org/html/2605.04539#A5.T11)\(Appendix[E](https://arxiv.org/html/2605.04539#A5)\)\.

### 4\.3Evaluation Tier

We evaluate on four metrics\.Textual Overlap: BLEU and BERTScore vs\. the student reference rationale\.Answer Anchoring: BERTScore\(Ans\) and Answer Coverage Rate \(ACR\) vs\. the correct\-option text\.Logical Entailment \(NLI\):SNLI=P​\(entail∣E,A\)∈\[0,1\]S\_\{\\text\{NLI\}\}\{=\}P\(\\text\{entail\}\\mid E,A\)\{\\in\}\[0,1\]fromcross\-encoder/nli\-deberta\-v3\-small\(premiseEE, hypothesisAA, joint max length512512, no prompt wrapping\)\.Verifier score \(Ver\): raw mean of the Alpaca\-7B Likert scorer of[bao2025exploring](https://arxiv.org/html/2605.04539#bib.bib16), reported as a diagnostic that the same automatic judge that produced the training signal does not penalise our outputs; the verifier is*not*used as a primary evaluation comparator\.

## 5Analysis and Results

We evaluateRLearner\-LLMagainst four classes of baseline: \(i\) the*SFT*starting checkpoint of each base architecture \(LLaMA\-2\-13B, Qwen3\-8B, Gemma 4 E4B\-it\), \(ii\) two of our own intermediate ablations*DPO v1*and*DPO v2*—verifier\-only DPO trained on165165and458458pairs respectively \(Cardiff/Sydney pools, no NLI signal\)—that surface the alignment tax that motivates the Hybrid design, \(iii\) the iterative\-refinement*ILearner\-LLM \(K=5\)*baseline of[bao2025exploring](https://arxiv.org/html/2605.04539#bib.bib16)reproduced from a LLaMA\-2\-13B SFT checkpoint with55refinement passes per question \(5×5\\timesthe inference compute of our single\-pass models\), and \(iv\) the proprietary*GPT\-4o\-mini*judge used in pairwise blind comparison\. Across the five\-domain test suite, RLearner\-LLM improves NLI entailment over its SFT baseline in1111of1515\(architecture, domain\) cells—the four non\-improvements are Qwen3 on Cardiff, Auckland Law, and Med Y1, and Gemma 4 on Sydney—surpasses the iterative ILearner\-LLM baseline on three of five domains, and wins95%95\\%of head\-to\-head pairwise comparisons against its own SFT baseline; on the same judge the framework loses95%95\\%to GPT\-4o\-mini’s longer outputs \(Section[5\.3](https://arxiv.org/html/2605.04539#S5.SS3), Table[3](https://arxiv.org/html/2605.04539#S5.T3)\)\. Table[1](https://arxiv.org/html/2605.04539#S5.T1)gives the headline picture; Sections[5\.1](https://arxiv.org/html/2605.04539#S5.SS1)–[5\.3](https://arxiv.org/html/2605.04539#S5.SS3)unpack each comparison\.

Table 1:NLI entailment improvement summary\. Each cell reportsSFT→RLearner\-LLM\\text\{SFT\}\\to\\text\{RLearner\-LLM\}scores; bold marks an improvement and the parenthesised number gives the relative gain\. RLearner\-LLM improves NLI in1111of1515cells\. The four non\-improvements are Qwen3 on Cardiff, Auckland Law, and Medicine Y1 \(domains where Qwen3\-8B’s SFT NLI is already comparatively high and the marginal gain shifts onto answer\-coverage—Qwen3 RLearner\-LLM increases ACR by\+8\+8to\+29\+29pp on every domain\), and Gemma 4 on Sydney Biology \(where SFT NLI of0\.24690\.2469is already the highest of any SFT in our study; full breakdown in Tables[2](https://arxiv.org/html/2605.04539#S5.T2)–[8](https://arxiv.org/html/2605.04539#A1.T8)\)\. The asterisk marks the first single\-pass\-RL result in our study to surpass the iterative ILearner\-LLM \(K=5\) baseline \(Gemma 4 on Auckland Law\)\.### 5\.1Comparison across Scientific Domains

Table[2](https://arxiv.org/html/2605.04539#S5.T2)gives the full per\-architecture breakdown on Cardiff Biology \(other Biology and Medicine domains in Appendix[A](https://arxiv.org/html/2605.04539#A1)\)\. On Cardiff, LLaMA\-2 RLearner lifts NLI5\.8×5\.8\\timesover SFT, Qwen3 trades a small NLI delta for a substantial ACR gain \(\+8\.5\+8\.5pp\), and Gemma 4 E4B\-it—despite4\.54\.5B effective parameters—achieves the highest NLI among RLearner variants \(0\.35050\.3505,\+65\.6%\+65\.6\\%over SFT\) at the fastest inference \(4\.764\.76s\)\. On Sydney, LLaMA\-2 reaches a6\.6×6\.6\\timeslift, Qwen3 gains\+31\.5%\+31\.5\\%NLI /\+23\.6\+23\.6pp ACR, and Gemma 4 SFT already sits at the highest SFT NLI in our study \(0\.24690\.2469\); we connect the flat Gemma 4 result to entailment\-headroom in §[6\.2](https://arxiv.org/html/2605.04539#S6.SS2)\.

Table 2:Experimental performance on the Cardiff Biology dataset across three base architectures\. LLaMA\-2 RLearner\-LLM achieves 5\.8×\\timesNLI improvement over its SFT baseline; Qwen3\-8B gains primarily in ACR under RL alignment; and Gemma 4 E4B\-it—despite having only 4\.5B effective parameters—achieves the highest NLI among RLearner variants \(0\.3505, \+65\.6% over its SFT baseline\) and the fastest inference \(4\.76s\), demonstrating that Hybrid\-DPO’s alignment\-tax mitigation transfers cleanly across architectures of substantially different scale and pre\-training recipe\. All Gemma 4 numbers use the same merged 5\-domain SFT protocol as Qwen3\-8B\.
### 5\.2Cross\-Domain Performance in Medicine and Law

Zero\-shot transfer of the same Hybrid\-DPO policy to Auckland Law and UK Medicine—neither contributed questions to the preference\-data construction—is summarised in Table[1](https://arxiv.org/html/2605.04539#S5.T1)\(per\-architecture detail in Appendix[A](https://arxiv.org/html/2605.04539#A1)\)\.Auckland Lawis the hardest single\-pass\-RL domain: LLaMA\-2 RLearner reaches0\.32290\.3229\(\+19\.5%\+19\.5\\%over SFT\) but falls short of ILearner\-LLM \(K=5\) at0\.39960\.3996, while Qwen3 delivers a\+28\.9\+28\.9pp ACR jump \(0\.5175→0\.80700\.5175\{\\to\}0\.8070\); most notably,RLearner\-LLM \(Gemma 4 E4B\-it\)is the first single\-pass\-RL result to surpass the ILearner\-LLM \(K=5\) benchmark on Auckland Law \(0\.43770\.4377vs0\.39960\.3996\)\. OnUK Medicine, all three architectures improve NLI; LLaMA\-2 peaks at0\.42510\.4251\(Y1\) and0\.38850\.3885\(Y2\), and Gemma 4 matches the LLaMA\-2 peak on Y2 \(0\.38920\.3892\) with∼\\sim3×\\timesfewer effective parameters\.

### 5\.3Pairwise Evaluation and the Persistent Verbosity Bias

Three pairwise comparisons \(Table[3](https://arxiv.org/html/2605.04539#S5.T3)\) probe LLM\-as\-a\-judge behaviour\. Row 1 replicates verbosity bias[wang2023large](https://arxiv.org/html/2605.04539#bib.bib11);[zheng2024judging](https://arxiv.org/html/2605.04539#bib.bib10);[park2024lengthbias](https://arxiv.org/html/2605.04539#bib.bib25): GPT\-4o\-mini awards69%69\\%to the verbose*LLaMA\-2 SFT*over the more concise, logically stronger*DPO v2*\. Row 2 reverses this pattern when both candidates come from the same family:RLearner\-LLM \(Qwen3\)wins95%95\\%vs\. its own Qwen3\-8B SFT\. Row 3 is the cautionary finding: against the proprietary*GPT\-4o\-mini*comparator, our concise RLearner\-LLM*loses*95%95\\%under the same GPT\-4o\-mini judge—we read this not as a quality verdict but as a continuation of the verbosity bias on a frontier comparator, since GPT\-4o\-mini’s longer hedged outputs satisfy the same length heuristic that drives Row 1\.

Table 3:Blind pairwise evaluation results judged by GPT\-4o\-mini \(n=100n\{=\}100per row, balanced left/right ordering\)\. The dagger \(†\\dagger\) on the first row marks our verbosity\-bias replication: the verbose SFT baseline is preferred over the more concise but logically more accurate DPO v2 baseline\. The third row reproduces the same bias on a frontier comparator: GPT\-4o\-mini’s longer outputs win95%95\\%of pairwise comparisons against our concise RLearner\-LLM under the same judge\.The juxtaposition of rows 2 and 3 is the headline takeaway: the alignment tax is a property of the*reward signal*used during preference\-data construction, not of DPO itself\. The Hybrid reward lets an88B local model beat its own SFT on the same judge, while the same judge reverts to verbosity bias against a frontier comparator—arguing for logic\-aware automatic metrics \(NLI, ACR\) over LLM\-as\-a\-judge as ground truth on logical quality\.

### 5\.4Reward\-Variant Ablation:HAH\_\{A\}vs\.HMH\_\{M\}

HMH\_\{M\}wins7/117/11NLI head\-to\-heads \(mean\+1\.5\+1\.5pp, Table[4](https://arxiv.org/html/2605.04539#S5.T4)\);HAH\_\{A\}wins the remaining44, all on heterogeneous cross\-domain pools\. Both variants beat SFT wherever the dual signal is supplied, supporting the view that the dual\-signal hypothesis—not the algebraic form—is the load\-bearing claim:HAH\_\{A\}vs\.HMH\_\{M\}is a hyperparameter chosen by pair\-pool feasibility \(§[3\.1](https://arxiv.org/html/2605.04539#S3.SS1)\)\.

Table 4:Head\-to\-headHAH\_\{A\}vs\.HMH\_\{M\}comparison on1111matched \(architecture, domain\) cells\. Bold marks the within\-row NLI winner\. Both variants beat the SFT baseline wherever the dual signal is supplied; their NLI difference is within∼1\{\\sim\}1pp on average, withHMH\_\{M\}winning7/117/11\.
### 5\.5SFT Failure Mode Analysis

A systematic error analysis on5,1005\{,\}100SFT\-generated explanations across the five domains \(Appendix[D](https://arxiv.org/html/2605.04539#A4)\) shows that standard SFT is fluent\-but\-vacuous in4848–85%85\\%of cases \(verbose withNLI<0\.05\\mathrm\{NLI\}\{<\}0\.05\), hallucinates citation\-style URLs in4444–69%69\\%, and fails to anchor the correct answer at all in1212–37%37\\%\(ACR=0\\mathrm\{ACR\}\{=\}0\)\. Hybrid\-DPO targets each failure mode by construction: the NLI signal in either variant \(HAH\_\{A\}orHMH\_\{M\}\) suppresses vacuous generations; underHMH\_\{M\}the ACR gate additionally eliminates answer\-evasive candidates and the length penalty discourages verbose hallucinations\. A qualitative case study \(Appendix[B](https://arxiv.org/html/2605.04539#A2)\) shows the same failure in microcosm: SFT lists facts but never commits to a deductive chain \(NLI=0\.001\\mathrm\{NLI\}\{=\}0\.001\), while RLearner\-LLM produces a structured proof on the same question \(NLI=0\.943\\mathrm\{NLI\}\{=\}0\.943,∼900×\\sim 900\\timesimprovement\)\.

## 6Discussion

### 6\.1Why the Hybrid Signal Works

NLI verifies thatEEimplies the correct option; the verifier captures clarity invisible to NLI\. Each alone degenerates \(answer\-repeating snippets or verbosity bias\), so the joint reward \(Eq\.[2](https://arxiv.org/html/2605.04539#S3.E2)\) filters both extremes and steers the policy to the alignment frontier’s upper\-right \(Fig\.[1](https://arxiv.org/html/2605.04539#S1.F1)\)\. The chosen/rejected NLI gap of1717–18×18\\timesalongside indistinguishable verifier scores replicates across all three architectures, confirming NLI as the discriminating signal and the verifier as a fluency regulariser\.Tier\-B robustness: a stricter Cardiff filter \(deleted=0,55\-star endorsement share≥0\.10\\geq 0\.10\) shrinks the pool7×7\\times\(7,309→1,0417\{,\}309\{\\to\}1\{,\}041\) yet*lifts*Hybrid\-DPO NLI by\+48%\+48\\%and ACR by\+13\.3\+13\.3pp, with the chosen/rejected NLI gap widening to18×18\\times\(Appendix[F](https://arxiv.org/html/2605.04539#A6), Table[12](https://arxiv.org/html/2605.04539#A6.T12)\)\.Tier\-C strict\-discriminator: using the full PeerWise answer\-submission logs \(now obtained for Cardiff and Sydney\), we additionally drop every question whose author key disagrees with the*modal student answer*—a conservative mis\-key/ambiguity proxy that was infeasible at submission time for lack of per\-option counts\. Stacked on Tier\-B, this strictest filter \(1,041→9551\{,\}041\{\\to\}955Cardiff,459→384459\{\\to\}384Sydney\) preserves the per\-architecture pattern of the main results: across all six architecture–corpus cells*every*cell improves ACR and four improve NLI\. The NLI lifts are largest on LLaMA\-2\-13B \(\+219%\+219\\%Cardiff,\+204%\+204\\%Sydney, from its lowest SFT baselines\) and strong on Gemma 4 E4B\-it \(\+109%\+109\\%/\+90%\+90\\%\), while Qwen3\-8B reproduces its default\-filter ACR\-up/NLI\-flat behaviour \(Appendix[G](https://arxiv.org/html/2605.04539#A7), Table[13](https://arxiv.org/html/2605.04539#A7.T13)\)\.

### 6\.2Architecture Effects and Entailment Headroom

Hybrid\-DPO’s marginal NLI gain is inversely related to the SFT NLI baseline\. LLaMA\-2\-13B SFT spans0\.050\.05–0\.270\.27across domains; RLearner\-LLM lifts these to0\.320\.32–0\.430\.43\(peak\+0\.339\+0\.339/4\.9×4\.9\\timeson UK Med Y1\)\. Qwen3\-8B SFT sits at0\.160\.16–0\.320\.32, so gains shift onto ACR \(\+9\+9to\+29\+29pp\)\. Gemma 4 E4B\-it \(4\.5B effective\) improves NLI on44of55domains—\+0\.229\+0\.229on UK Med Y2 \(2\.4×2\.4\\times\) matches the LLaMA\-2 peak with∼\\sim3×\\timesfewer effective parameters—and is the first single\-pass result to surpass ILearner\-LLM \(K=5\) on Auckland Law\. The non\-improvement is Sydney Biology, where SFT NLI \(0\.24690\.2469\) is already the highest in our study\. These patterns suggest Hybrid\-DPO tracks*entailment headroom*rather than raw model scale\. On three of the four Gemma 4 NLI\-gain domains, NLI rises with a modest ACR reduction \(−10\.3\-10\.3to−1\.3\-1\.3pp on Cardiff, Med Y1, Med Y2; Auckland Law gains both,\+11\.9%\+11\.9\\%/\+3\.9\+3\.9pp\);HMH\_\{M\}’s ACR gate is a floor, not a ceiling, so a stronger ACR weight is a natural extension to recover Qwen3’s ACR profile\.

### 6\.3Limitations

Three limitations apply\. \(i\) LLaMA\-2 on Auckland Law \(NLI0\.32290\.3229\) is surpassed by ILearner\-LLM \(K=5\) at0\.39960\.3996\(Gemma 4 closes the gap, §[6\.2](https://arxiv.org/html/2605.04539#S6.SS2)\); adding targeted iteration is future work\. \(ii\)Circular\-evaluation risk: NLI is reported using the samedeberta\-v3\-smallthat scores training pairs\. Three properties mitigate this—dual\-signal coupling,HMH\_\{M\}’s independent ACR gate, and out\-of\-family GPT\-4o\-mini pairwise judgements \(Table[3](https://arxiv.org/html/2605.04539#S5.T3)\)—but a held\-out, larger NLI model \(DeBERTa\-v3\-large or RoBERTa\-large MNLI\) would confirm the∼\\sim1–3 pp configuration gaps and yield more discriminative preference labels\. \(iii\) The PeerWise SFT corpus is undergraduate\-authored; partial justifications cap overlap\-style metrics, motivating an expert\-curated rerun\.

## 7Conclusion

RLearner\-LLMresolves DPO’s reward\-signal blindspot via a dual\-signal Hybrid reward \(NLI\+\+verifier, additiveHAH\_\{A\}or multiplicative\-ACRHMH\_\{M\}by pair\-count feasibility\)\. Across55domains and33architectures, Hybrid\-DPO improves NLI in11/1511/15cells \(up to6×6\\timesover SFT\), is the first single\-pass\-RL result to surpass ILearner\-LLM \(K=5\) on Auckland Law, and wins95%95\\%vs\. its own SFT; the symmetric95%95\\%loss to GPT\-4o\-mini’s longer outputs under the same judge replicates verbosity bias on a frontier comparator, arguing for logic\-aware metrics over LLM\-as\-a\-judge in knowledge\-intensive generation\.

## References

- \(1\)Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L\. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe\.Training language models to follow instructions with human feedback\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 35, pages 27730–27744, 2022\. arXiv:2203\.02155\.
- \(2\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\. arXiv:1707\.06347, 2017\.
- \(3\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 36, 2023\. arXiv:2305\.18290\.
- \(4\)Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen\-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela\.Retrieval\-augmented generation for knowledge\-intensive NLP tasks\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 33, pages 9459–9474, 2020\. arXiv:2005\.11401\.
- \(5\)Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins\.Solving math word problems with process\- and outcome\-based feedback\. arXiv:2211\.14275, 2022\.
- \(6\)Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang\.RRHF: Rank responses to align language models with human feedback without tears\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 36, 2023\. arXiv:2304\.05302\.
- \(7\)Jiwoo Hong, Noah Lee, and James Thorne\.ORPO: Monolithic preference optimization without reference model\. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\), pages 11170–11189, 2024\. arXiv:2403\.07691\.
- \(8\)Yu Meng, Mengzhou Xia, and Danqi Chen\.SimPO: Simple preference optimization with a reference\-free reward\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 37, 2024\. arXiv:2405\.14734\.
- \(9\)Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He\.β\\beta\-DPO: Direct preference optimization with dynamicβ\\beta\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 37, 2024\. arXiv:2407\.08639\.
- \(10\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\. In Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track, volume 36, 2023\. arXiv:2306\.05685\.
- \(11\)Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui\.Large language models are not fair evaluators\. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\), pages 9440–9450, 2024\. arXiv:2305\.17926\.
- \(12\)Arjun Panickssery, Samuel R\. Bowman, and Shi Feng\.LLM evaluators recognize and favor their own generations\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 37, 2024\. arXiv:2404\.13076\.
- \(13\)Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T\. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li\.DecodingTrust: A comprehensive assessment of trustworthiness in GPT models\. In Advances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track \(Outstanding Paper\), volume 36, 2023\. arXiv:2306\.11698\.
- \(14\)Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo\.Are emergent abilities of large language models a mirage?In Advances in Neural Information Processing Systems \(NeurIPS\), volume 36, 2023\. arXiv:2304\.15004\.
- \(15\)Miriam Rateike, Celia Cintas, John Wamburu, Tanya Akumu, and Skyler Speakman\.Weakly supervised detection of hallucinations in LLM activations\. In NeurIPS 2023 Workshop on Socially Responsible Language Modelling Research \(SoLaR\), 2023\. arXiv:2312\.02798\.
- \(16\)Qiming Bao, Juho Leinonen, Alex Yuxuan Peng, Wanjun Zhong, Gael Gendron, Timothy Pistotti, Alice Huang, Paul Denny, Michael Witbrock, and Jiamou Liu\.Exploring iterative enhancement for improving learnersourced multiple\-choice question explanations with large language models\. In Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI/EAAI\-25\), pages 28955–28963, 2025\. arXiv:2309\.10444\.
- \(17\)Dongkyu Cho, Amy B\. Z\. Zhang, Bilel Fehri, Sheng Wang, Rumi Chunara, Hengrui Cai, and Rui Song\.Correct reasoning paths visit shared decision pivots\. In NeurIPS 2025 Workshop on Foundations of Reasoning in Language Models \(FoRLM\), 2025\. arXiv:2509\.21549\.
- \(18\)Zixuan Huang, Yikun Ban, Lean Fu, Xiaojie Li, Zhongxiang Dai, Jianxin Li, and Deqing Wang\.Adaptive batch\-wise sample scheduling for direct preference optimization\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 38, 2025\. arXiv:2506\.17252\.
- \(19\)Botong Zhang, Shuo Li, Ignacio Hounie, Osbert Bastani, Dongsheng Ding, and Alejandro Ribeiro\.Alignment of large language models with constrained learning\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 38, 2025\. arXiv:2505\.19387\.
- \(20\)Chenyu Zhu, Yefeng Liu, Hao Zhang, Aowen Wang, Yangxue, Guanhua Chen, Longyue Wang, Weihua Luo, and Kaifu Zhang\.Alleviating hallucinations in large language models through multi\-model contrastive decoding and dynamic hallucination detection\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 38, 2025\.
- \(21\)Brown Ebouky, Andrea Bartezzaghi, and Mattia Rigotti\.Eliciting reasoning in language models with cognitive tools\. In Advances in Neural Information Processing Systems \(NeurIPS\), volume 38, 2025\. arXiv:2506\.12115\.
- \(22\)Borong Zhang, Yuhao Zhang, Jiaming Ji, Yingshan Lei, Yishuai Cai, Josef Dai, Yuanpei Chen, and Yaodong Yang\.SafeVLA: Towards safety alignment of vision\-language\-action model via constrained learning\. In Advances in Neural Information Processing Systems \(NeurIPS\), Spotlight, volume 38, 2025\. arXiv:2503\.03480\.
- \(23\)Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, Tianyi Qiu, and Yaodong Yang\.Aligner: Efficient alignment by learning to correct\. In Advances in Neural Information Processing Systems \(NeurIPS\), 2024\. arXiv:2402\.02416\.
- \(24\)Haozhe Ma, Kuankuan Sima, Thanh Vinh Vo, Di Fu, and Tze\-Yun Leong\.Reward shaping for reinforcement learning with an assistant reward agent\. In International Conference on Machine Learning \(ICML\), 2024\.
- \(25\)Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn\.Disentangling length from quality in direct preference optimization\. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4998–5017, 2024\. arXiv:2403\.19159\.

## Appendix APer\-Domain Detail Tables

For completeness we provide the full per\-architecture breakdown on each of the four non\-Cardiff domains\. The Cardiff Biology table \(Table[2](https://arxiv.org/html/2605.04539#S5.T2)\) appears in the main paper because it is the most extensively dissected domain and is the target of the robustness ablation in Section[6\.3](https://arxiv.org/html/2605.04539#S6.SS3); the headline NLI summary across all five domains is given in Table[1](https://arxiv.org/html/2605.04539#S5.T1)\.

Table 5:Sydney Biology, full per\-architecture breakdown\.Table 6:Auckland Law, full per\-architecture breakdown\. Gemma 4 E4B\-it RLearner surpasses the iterative ILearner\-LLM \(K=5\) NLI benchmark of0\.39960\.3996as the first single\-pass RL method in our study\.Table 7:UK Medicine Year 1, full per\-architecture breakdown\.Table 8:UK Medicine Year 2, full per\-architecture breakdown\. Gemma 4 RLearner virtually matches the LLaMA\-2\-13B peak \(0\.38920\.3892vs0\.38850\.3885\) with∼3×\\sim 3\\timesfewer effective parameters\.
## Appendix BQualitative Case Study

To complement the quantitative results of Section[5](https://arxiv.org/html/2605.04539#S5), we provide a representative qualitative comparison of an SFT baseline output against a RLearner\-LLM output on a Cardiff Biology biochemistry question\. The example illustrates the failure mode that the Hybrid reward is designed to correct: SFT possesses the relevant facts but does not commit to a deductive chain that justifies the correct option, while RLearner\-LLM produces a structured proof and consequently scores three orders of magnitude higher on NLI entailment\.

Question:Which of the following is TRUE during a period of high intensity exercise \(e\.g\., sprinting\)?Options:\(A\) Oxygen is consumed during glycolysis; \(B\) Oxygen rate measures energy expenditure; \(C\)ATP is generated anaerobically through substrate level phosphorylation using creatine phosphate; \(D\) CO2 rate measures energy\.SFT \(Baseline\) Output:A is incorrect as glycolysis is an anaerobic process and does not require oxygen\. B is incorrect as during high intensity exercise THE rate of oxygen consumption is not a direct measure… \[Truncated: Logic drifts into redundant negation\]\.NLI Score: 0\.001RLearner\-LLM \(Ours\) Output:During high\-intensity exercise such as sprinting, the body’s demand for energy is extremely high and cannot be met by aerobic pathways alone\.Option C is correctbecause ATP is generated anaerobically through substrate\-level phosphorylation using creatine phosphate\. Let’s analyze: Option A is incorrect because Glycolysis is anaerobic and does not consume oxygen… \[Detailed logical derivation\]\.NLI Score: 0\.943Table 9:Qualitative case study on a Cardiff Biology biochemistry question: the SFT baseline \(LLaMA\-2\-13B\) provides correct facts but fails to commit to a deductive chain that justifies the correct option, yielding an NLI entailment score near zero\. RLearner\-LLM provides a structured proof, increasing NLI by roughly900×900\\times\.
## Appendix CImplementation Notes for Multimodal Base Models

This appendix gives the base\-model\-specific LoRA target details that were summarised in Section 3\.

Qwen3\-8B\.Qwen3 applies RMSNorm toqqandkk*after*the projection reshape \(QK\-norm\)\. LoRA adapters placed onq\_projork\_projintroduce shape\-incompatible tensors into this normalisation path, triggering a CUBLAS error at the first attention layer\. We therefore restrict the Qwen3 LoRA target set tov\_proj,o\_proj,gate\_proj,up\_proj,down\_proj\.

Gemma 4 E4B\-it\.Gemma 4 ships as a multimodalGemma4ForConditionalGenerationcheckpoint in which the text\-tower weights live under themodel\.language\_model\.∗\*prefix\. Loading withAutoModelForCausalLMsilently random\-initialises the text tower because the prefix does not match\. In addition, the vision and audio towers wrap their attention projections inside a customGemma4ClippableLinearclass that PEFT cannot adapt; using leaf\-name LoRA targets such asq\_projtherefore inadvertently matches the vision and audio wrappers and aborts adapter injection\. We resolve both issues by instantiatingGemma4ForConditionalGenerationexplicitly \(the vision and audio towers remain present but idle during text\-only generation\) and specifying LoRA target modules as a full\-path regular expression that restricts adaptation to the text tower’s plainnn\.Linearprojections:

\.\*language\_model\.layers\.\\d\+\.\(self\_attn\.\(q\|k\|v\|o\)\_proj\|mlp\.\(gate\|up\|down\)\_proj\)$This pattern matches exactly258258Linear modules \(4242layers×\\times\{q,o,gate,up,down\}\\\{q,o,\\text\{gate\},\\text\{up\},\\text\{down\}\\\}\+2424layers×\\times\{k,v\}\\\{k,v\\\}; Gemma 4 uses unifiedkk/vvin local\-sliding layers\)\. Because Gemma 4 has no in\-attention QK\-norm, all four attention projections can be LoRA\-adapted safely\. A practical caveat: Gemma 4’s vocabulary size of262262K inflates its cross\-entropy baseline tolog⁡262,144≈12\.5\\log 262\{,\}144\\approx 12\.5nats, so raw SFT losses of99–1010are normal and are better interpreted viamean\_token\_accuracyor generation sanity checks than by the loss magnitude alone\.

Multi\-stage adapter composition\.When evaluating the RL\-aligned policy, the DPO LoRA adapter must be applied on top of the*SFT\-merged*base weights, since the DPO delta was trained relative to that reference point\. Loading the DPO adapter directly on the raw base produces a model whose behaviour reverts toward the base instruction\-tuning distribution; the correct inference pipeline isbase→apply SFT→merge→apply DPO→merge\\text\{base\}\\to\\text\{apply SFT\}\\to\\text\{merge\}\\to\\text\{apply DPO\}\\to\\text\{merge\}\.

## Appendix DSFT Failure Mode Analysis

For completeness we report the per\-domain SFT failure\-mode rates referenced in Section[5\.5](https://arxiv.org/html/2605.04539#S5.SS5)\. Each metric is a deterministic function of the generated text: Answer Evasion =𝕀​\[ACR=0\]\\mathbb\{I\}\[\\mathrm\{ACR\}\{=\}0\]; Verbose\-low\-NLI =𝕀​\[\|E\|chars\>400∧NLI<0\.05\]\\mathbb\{I\}\[\|E\|\_\{\\text\{chars\}\}\{\>\}400\\land\\mathrm\{NLI\}\{<\}0\.05\]; Hallucinated URLs = regex match forhttps?://\\S\+\|www\\\.\\S\+; Cyclic Repetition = any66\-gram repeated\.

Table 10:Per\-domain SFT failure\-mode rates \(5,100 SFT\-generated explanations, LLaMA\-2\-13B\)\. Standard SFT produces fluent but logically vacuous outputs in4848–85%85\\%of cases \(Verbose\-low\-NLI\), hallucinates citation\-style URLs in4444–69%69\\%, and fails to anchor the correct answer at all in1212–37%37\\%of cases \(Answer Evasion\)\. These rates motivate the Hybrid reward’s combination of an NLI signal, an ACR gate, and a length penalty\.
## Appendix EPer\-Architecture Reward Variant Assignment

Table[11](https://arxiv.org/html/2605.04539#A5.T11)records which Hybrid reward variant—HAH\_\{A\}\(the additive cross\-domain form\) orHMH\_\{M\}\(the multiplicative single\- or merged\-domain form\)—was used for each \(architecture, domain\) cell, together with the number of preference pairs constructed for each\.

Table 11:Hybrid reward variant used per \(architecture, domain\) cell, with preference\-pair counts\. A head\-to\-headHAH\_\{A\}vs\.HMH\_\{M\}ablation appears in Table[4](https://arxiv.org/html/2605.04539#S5.T4)\(main text\)\.
## Appendix FTier\-B Robustness Ablation Detail

Table[12](https://arxiv.org/html/2605.04539#A6.T12)details the Tier\-B robustness ablation on Cardiff Biology: a stricter question\-quality filter \(deleted=0 plus a55\-star endorsement share≥0\.10\\geq 0\.10\) shrinks the training pool7×7\\times\(from7,3097\{,\}309to1,0411\{,\}041questions\), yet Hybrid\-DPO trained on the smaller corpus improves on every metric\.

Table 12:Cardiff Biology robustness to question\-quality filtering on theGemma 4 E4B\-itarchitecture \(the SFT NLI of0\.21170\.2117identifies the row\)\. Tier\-B addsdeleted=0 and a55\-star endorsement share≥0\.10\\geq 0\.10to the default filter, shrinking the Cardiff training corpus by7\.0×7\.0\\times\. The SFT baseline is essentially unchanged \(NLI0\.2117→0\.20590\.2117\\to 0\.2059\), but Hybrid\-DPO trained on the smaller Tier\-B corpus reaches*higher*NLI \(0\.3505→0\.52020\.3505\\to 0\.5202\), ACR \(0\.6497→0\.78230\.6497\\to 0\.7823\), and answer\-anchored BERTScore than under the default filter, with comparable BLEU and BERT\(Stu\)\. The rightmost column reports per\-question inference time in seconds\.
## Appendix GTier\-C Strict\-Discriminator Ablation Detail

The original submission noted that the*strict\-discriminator*variant of the question\-quality filter—retaining only questions whose author\-designated key coincides with the*modal student answer*—requires per\-option response counts absent from the question\-level metadata\. We have since obtained the full PeerWise answer\-submission logs for both Cardiff and Sydney \(3,439,3403\{,\}439\{,\}340and503,806503\{,\}806individual submissions respectively\), enabling this variant\. For each question we compute the most\-frequently\-selected option across all student submissions anddrop questions whose author key disagrees with this modal answer, a conservative proxy for mis\-keyed or ambiguous items\. Stacked on top of the Tier\-B filter, this definesTier\-C, the strictest filter in our study; it further shrinks the Cardiff pool1,041→9551\{,\}041\\to 955and the Sydney pool459→384459\\to 384\.

Table 13:Robustness of Hybrid\-DPO to theTier\-C strict\-discriminator filter\(Tier\-B\+\+author\-key=\{=\}modal\-student\-answer\), evaluated per\-architecture on Cardiff and Sydney\. Each model’s DPO row is compared against its own SFT baseline on the same Tier\-C test split \(NLI / ACR are not comparable across filter tiers, which use different held\-out sets\)\. Under the strictest filter, Hybrid\-DPO preserves a large NLI lift on Gemma 4 E4B\-it \(\+109%\+109\\%Cardiff,\+90%\+90\\%Sydney\) and raises ACR on*every*architecture–corpus cell\. Qwen3\-8B reproduces the ACR\-up/NLI\-flat pattern it exhibits under the default filter \(cf\. Table[2](https://arxiv.org/html/2605.04539#S5.T2)\), confirming that the strict filter preserves rather than distorts the per\-architecture behaviour of the main results\. LLaMA\-2\-13B shows the largest NLI lifts \(\+219%\+219\\%Cardiff,\+204%\+204\\%Sydney\), consistent with its lowest SFT baselines and the entailment\-headroom effect of §[6\.2](https://arxiv.org/html/2605.04539#S6.SS2)\. Across all six architecture–corpus cells,44improve NLI and*all*improve ACR under the strictest filter\.
## Appendix HHuggingFace Identifiers and Model Versions

For exact reproducibility we list the HuggingFace identifiers, parameter counts, and the precise checkpoint version used for every model in the pipeline\. All weights were resolved against the HuggingFace Hub at the start of training; loading viatransformerswith the listed identifier reproduces our results bit\-for\-bit when combined with the released LoRA adapters and preference data\.

Table 14:HuggingFace identifiers and parameter counts for every model used in the pipeline\. The Alpaca\-7B verifier is a domain\-finetuned checkpoint released by[bao2025exploring](https://arxiv.org/html/2605.04539#bib.bib16); we use the exact merged\-corpus weights distributed with that paper\. The pairwise judge runs againstgpt\-4o\-mini\-2024\-07\-18via the OpenAI API at temperature0for deterministic adjudication\.
## Appendix ILLM Judge Prompt and Decoding Settings

The pairwise judge results in Table[3](https://arxiv.org/html/2605.04539#S5.T3)\(Section[5\.3](https://arxiv.org/html/2605.04539#S5.SS3)\) use the following prompt template, applied to every \(question, context,AA,BB\) tuple at temperature0:

> You are an expert evaluator of AI\-generated educational explanations\. Given a question and its context, compare two provided explanations and decide which one is better for a student\. Consider the following criteria: 1\. Accuracy: Is the explanation factually correct? 2\. Soundness: Is the reasoning logical and easy to follow? 3\. Helpfulness: Does it truly help a student understand WHY the answer is correct? Question:\{question\} Context:\{context\} Explanation 1:\{choice\_a\} Explanation 2:\{choice\_b\} Which explanation is better? Provide a very brief justification followed by your choice in the format: “Better: Explanation \[1/2/Tie\]”\.

#### Order randomisation\.

For each test item we run the judge twice with the order of Explanation 1 and Explanation 2 swapped, then average the two verdicts; this controls for the well\-documented position bias[wang2023large](https://arxiv.org/html/2605.04539#bib.bib11)\. The100100items reported per row in Table[3](https://arxiv.org/html/2605.04539#S5.T3)therefore cost200200judge calls per row\.

#### Decoding settings\.

model="gpt\-4o\-mini\-2024\-07\-18",temperature=0\.0,max\_tokens=200, all other parameters at OpenAI defaults\. Verdicts are extracted by string match on"Better: Explanation 1"/"Better: Explanation 2"; everything else \(including malformed outputs\) is counted as a tie\.

#### NLI verifier inputs\.

Thecross\-encoder/nli\-deberta\-v3\-smallscorer takes the explanationEEas the premise and the correct\-option textAAas the hypothesis, jointly tokenised at maximum length512512, with no prompt wrapping \(Section[4\.3](https://arxiv.org/html/2605.04539#S4.SS3)\)\. The Alpaca\-7B stylistic verifier uses the standard Alpaca instruction\-input\-response template with the instruction‘‘As a question rating verifier expert, can you generate the question rating score for the given input?’’; the score is parsed as the first\{1,2,3,4,5\}digit in the response\.