QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides

# QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides
Source: [https://arxiv.org/html/2607.15810](https://arxiv.org/html/2607.15810)
Zhengyang ZhugeHao Yu††footnotemark:Xin Wang††footnotemark:Zheng LiYizhong CaoDayiheng LiuJianwei Zhang Qwen Team, Alibaba Inc

###### Abstract

Rollout generation is a major bottleneck in Reinforcement Learning \(RL\) for Mixture\-of\-Experts \(MoE\) Large Language Models, motivating low\-precision rollout acceleration such as FP8\. As an emerging low\-precision format, NVFP4 combines fine\-grained scaling for accuracy preservation with native W4A4 FP4 GEMMs for higher throughput than FP8\. However, we find that directly applying NVFP4 to MoE RL rollout is impractical\. NVFP4 rollout with BF16 training collapses after roughly 150 steps, accompanied by rapidly growing rollout–trainer log\-probability gaps\. Through training–inference error analysis and controlled ablations, we identify activation error, rather than weight error, as the dominant source of FP4 RL instability: weights can be synchronized and aligned by a shared quantization–dequantization path, whereas activations are recomputed online and error is amplified by the coarse E2M1 grid\. Therefore, to stabilize NVFP4 RL for MoE, we proposeQUantization\-errorAlignment acrossDualSides \(QUADS\)\. On the trainer side, we introduce Asymmetric Quantization\-Aware Training fake\-quantizing weights while keeping activations unquantized for better alignment\. On the rollout side, Residual Activation Compensation corrects high\-error activation channels while preserving native W4A4 GEMMs\. In our MoE RL experiments on several benchmarks, QUADS achieves BF16\-level accuracy, improves average pass@1 by 21\.49 points over naive NVFP4 RL, and delivers∼\\sim16% higher rollout throughput than FP8\.

## 1Introduction

Reinforcement learning \(RL\) has become a central paradigm for scaling the reasoning capabilities of Large Language Models \(LLMs\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.15810#bib.bib5)\)\. Modern pipelines decouple autoregressive*rollout*on an inference engine from gradient computation on a separate trainer, yet rollout generation still dominates end\-to\-end wall\-clock time—often exceeding 70% of each training step\(Liet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib9); Xiet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib11); Qiuet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib12)\)\. As models generate longer chain\-of\-thought traces, this imbalance intensifies, making rollout acceleration a first\-order systems priority for policy\-gradient methods such as GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.15810#bib.bib5)\)\. The challenge is that rollout precision is not merely a serving detail: token\-level log\-probabilities produced by the rollout engine determine importance ratios in the trainer, so numerical bias can directly corrupt the policy gradient\.

A common acceleration strategy is to run rollout inference at reduced precision\. At the FP8 tier, recent RL systems have shown that the resulting*training–inference precision mismatch*can often be controlled with truncated importance sampling \(TIS\)\(Yaoet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib7)\), adaptive clipping, or unified FP8 precision flows\(Liet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib9); Xiet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib11); Qiuet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib12); Zhouet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib13)\)\. This makes FP8 a strong practical baseline: it accelerates rollout while keeping the log\-probability gap small enough for stable optimization\.

Recently, NVIDIA Blackwell introduces a more aggressive target\. NVFP4 enables native*W4A4*general matrix multiplications \(GEMMs\) on FP4 Tensor Cores, with peak GEMM throughput up to∼\\sim2×\\timeshigher than FP8 and∼\\sim4×\\timesover BF16\(Alvarezet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib1); Jarmusch and Chandrasekaran,[2025](https://arxiv.org/html/2607.15810#bib.bib3); NVIDIA,[2025](https://arxiv.org/html/2607.15810#bib.bib2)\)\. However, NVFP4 is qualitatively less forgiving than FP8: its E2M1 core has only eight positive representable levels per sign, compared with 256 for FP8 E4M3, and true W4A4 execution quantizes activations on every rollout forward pass\. As shown in Figure[1](https://arxiv.org/html/2607.15810#S1.F1), directly combining NVFP4 W4A4 rollout with BF16 training collapses within roughly 150 steps: reward and held\-out score both fall while the per\-token log\-probability gap grows monotonically, pushing importance ratios outside the trust region\.

To identify the source of this failure, we conduct a motivating ablation \(Section[3\.1](https://arxiv.org/html/2607.15810#S3.SS1)\) that independently quantizes weights and activations during rollout\. Comparing W4A16 \(FP4 weights, BF16 activations\) with W16A4 \(BF16 weights, FP4 activations\) reveals a strong operand asymmetry:*activation quantization*, not weight quantization, is the dominant driver of log\-probability divergence and reward collapse\. W16A4 closely tracks the failure mode of full W4A4, whereas W4A16 remains much closer to stable behavior; the two error sources are approximately additive\. This reframes NVFP4 RL as a quantization error alignment problem\.

Existing approaches do not resolve this setting\. Weight\-only NVFP4 methods such as QeRL\(Huanget al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib8)\)keep activations in BF16 and execute FP16 GEMMs instead of native W4A4 FP4 Tensor Core operations, while FP8 and INT8 based RL frameworks\(Liet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib9); Guet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib10); Xiet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib11); Qiuet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib12); Zhouet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib13)\)operate on finer numerical grids where rollout–trainer mismatch is more tractable\. Stable full\-parameter MoE GRPO with true W4A4 rollout on Blackwell therefore still requires alignment strategies that directly target the activation\-dominated mismatch identified above\.

We address this gap with QUantization\-error Alignment across Dual Sides \(QUADS\) for NVFP4 RL\. On the training side, asymmetric W4A16 quantization\-aware training aligns the weight quantization–dequantization path while keeping learner activations in BF16 \(Section[3\.2](https://arxiv.org/html/2607.15810#S3.SS2)\)\. This design exploits the fact that weights are synchronized across engines and can therefore share a matched QDQ path, while activations are recomputed online and cannot be reliably aligned by symmetric W4A4 fake quantization\. On the rollout side, residual activation compensation reduces the remaining rollout\-time activation error through a second\-pass quantization of high\-residual channels, implemented with fused kernels to preserve most of the W4A4 throughput advantage \(Section[3\.3](https://arxiv.org/html/2607.15810#S3.SS3)\)\. Together, these components recover BF16\-level accuracy in full\-parameter MoE GRPO: our QUADS pipeline reaches 72\.86% average pass@1 across four held\-out benchmarks, matching the 73\.15% BF16 baseline and avoiding the severe degradation of naive NVFP4 training \(51\.37%\)\. At the same time, it improves rollout throughput by 16% over FP8\.

Our main contributions are:

- •NVFP4 RL failure diagnosis\.Through ablations, we show that activation quantization is essential in NVFP4 RL stability and explains the collapse of naive W4A4 RL\.
- •Asymmetric Quantization\-Aware\-Training\.On the trainer\-side, we propose the asymmetric QAT scheme that fake\-quantizes weights to NVFP4 but leaves activations unquantized, reducing weight mismatch without amplifying cross\-engine activation drift\.
- •Residual Activation Compensation\.On the rollout\-side, we further reduce activation error through second\-pass quantization of high\-residual channels, preserving native W4A4 GEMMs while closing the remaining log\-probability gap\.
- •NVFP4 RL empirical validation for MoE\.On NVFP4 RL training of MoE model, our proposed QUADS pipeline matches BF16\-level benchmark accuracy and retains a throughput advantage over FP8 rollout\.

![Refer to caption](https://arxiv.org/html/2607.15810v1/x1.png)

\(a\) Training Reward

![Refer to caption](https://arxiv.org/html/2607.15810v1/x2.png)

\(b\) Test Score

![Refer to caption](https://arxiv.org/html/2607.15810v1/x3.png)

\(c\) Log\-Prob Diff

Figure 1:Naive NVFP4 collapse in RL\.\(a\)Mean training reward: naive NVFP4 W4A4 rollout with BF16 training \(blue\) peaks near step 150 and then collapses, whereas the BF16 RL baseline \(red\) continues to improve steadily\.\(b\)Held\-out test score shows the same pattern, confirming that the failure is not limited to the training reward signal\.\(c\)Log\-probability difference between the NVFP4 rollout engine and the BF16 trainer diverges extremely, indicating a severe training–inference mismatch that importance sampling cannot correct\.
## 2Preliminaries

### 2\.1Reinforcement Learning for LLMs

Modern LLM post\-training increasingly relies on Reinforcement Learning with Verifiable Rewards \(RLVR\)\. A typical pipeline decouples*rollout*—autoregressive response generation on a throughput\-oriented inference engine, from*training*—gradient computation on a separate backend such as FSDP or Megatron\. Policy\-gradient algorithms update an autoregressive policyπθ\\pi\_\{\\theta\}using samples drawn from a reference policyπref\\pi\_\{\\mathrm\{ref\}\}\. Throughout this work we adopt GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.15810#bib.bib5)\)as the representative algorithm and analyze how numerical precision in rollout affects its stability\.

##### GRPO

Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.15810#bib.bib5)\)extends PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.15810#bib.bib4)\)by eliminating the value model: for each prompt, it samples a group ofGGresponses\{oi\}i=1G\\\{o\_\{i\}\\\}\_\{i=1\}^\{G\}from the old policyπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}and normalizes their scalar rewards within the group to obtain advantagesA^i,t\\hat\{A\}\_\{i,t\}\. The clipped surrogate objective is

𝒥GRPO​\(θ\)=𝔼​\[1G​∑i=1G1\|oi\|​∑t=1\|oi\|min⁡\(ri,t​\(θ\)​A^i,t,clip​\(ri,t​\(θ\),1−ϵ,1\+ϵ\)​A^i,t\)−β​DKL​\(πθ∥πref\)\],\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|o\_\{i\}\|\}\\sum\_\{t=1\}^\{\|o\_\{i\}\|\}\\min\\\!\\Bigl\(r\_\{i,t\}\(\\theta\)\\,\\hat\{A\}\_\{i,t\},\\;\\mathrm\{clip\}\\bigl\(r\_\{i,t\}\(\\theta\),\\,1\-\\epsilon,\\,1\+\\epsilon\\bigr\)\\,\\hat\{A\}\_\{i,t\}\\Bigr\)\-\\beta\\,D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\mathrm\{ref\}\}\)\\right\],\(1\)whereri,t​\(θ\)r\_\{i,t\}\(\\theta\)is the per\-token importance ratio \(Eq\. \([2](https://arxiv.org/html/2607.15810#S2.E2)\)\),ϵ\\epsilonbounds the trust region, andβ\\betacontrols the KL penalty againstπref\\pi\_\{\\mathrm\{ref\}\}\. Both the clipped surrogate and the KL term depend on token\-level log\-probabilities; biased estimates therefore directly corrupt the policy gradient\.

##### Training–Inference Mismatch

When rollout and training share the same engine and numerical path, GRPO reduces to standard on\-policy optimization\. In practice, however, responses are sampled by an inference policyπsampler\\pi\_\{\\mathrm\{sampler\}\}\(e\.g\., vLLM or SGLang\) while gradients are computed with a training policyπlearner\\pi\_\{\\mathrm\{learner\}\}\(e\.g\., FSDP\)\(Yaoet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib7)\)\. Importance sampling \(IS\) corrects this off\-policy gap: for each tokenata\_\{t\}given contextsts\_\{t\}, the per\-token ratio is

rt​\(θ\)=πθ​\(at∣st\)πθold​\(at∣st\)=exp⁡\(log⁡πθ​\(at∣st\)−log⁡πθold​\(at∣st\)\)\.r\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\)\}=\\exp\\\!\\bigl\(\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\-\\log\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\)\\bigr\)\.\(2\)We denote the log\-probability differenceδt≜log⁡πθ​\(at∣st\)−log⁡πθold​\(at∣st\)\\delta\_\{t\}\\triangleq\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\-\\log\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\)\. Whenπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}andπθ\\pi\_\{\\theta\}are evaluated at different precisions or through different kernel paths,δt≠0\\delta\_\{t\}\\neq 0even for identical parameters—this is the*training–inference mismatch*\(Qiet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib6); Guet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib10); Qiuet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib12)\)\.

Low\-precision rollout amplifies the problem\. In FP8 settings the resulting gaps are comparatively modest and can often be controlled with truncated importance sampling \(TIS\)\(Yaoet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib7)\), adaptive clipping\(Liet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib9); Zhouet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib13)\), or unified FP8 precision flows\(Xiet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib11); Qiuet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib12)\)\. Under NVFP4 W4A4 rollout with BF16 training, by contrast,\|δt\|\|\\delta\_\{t\}\|grows rapidly over optimization steps, importance ratios leave\[1−ϵ,1\+ϵ\]\[1\-\\epsilon,\\,1\+\\epsilon\], and reward curves collapse \(Figure[1](https://arxiv.org/html/2607.15810#S1.F1)\)\.

We analyze mismatch through the decomposition

δt≈δtweight\+δtact\+δtnum,\\delta\_\{t\}\\;\\approx\\;\\delta\_\{t\}^\{\\mathrm\{weight\}\}\+\\delta\_\{t\}^\{\\mathrm\{act\}\}\+\\delta\_\{t\}^\{\\mathrm\{num\}\},\(3\)whereδtweight\\delta\_\{t\}^\{\\mathrm\{weight\}\}arises from weight quantization,δtact\\delta\_\{t\}^\{\\mathrm\{act\}\}from activation quantization, andδtnum\\delta\_\{t\}^\{\\mathrm\{num\}\}from residual numerical\-path differences between engines\(Qiet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib6); Yaoet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib7)\)\. This decomposition guides our dual\-side alignment strategy in Section[3](https://arxiv.org/html/2607.15810#S3)\.

### 2\.2Low\-Precision Inference

Because rollout dominates RL wall\-clock time, a standard acceleration strategy is to execute linear\-layer GEMMs at reduced precision while keeping gradient computation in BF16 or FP32\. Quantization maps a high\-precision tensorXXto a low\-precision surrogateX^\\hat\{X\}via a scale factor and a discrete rounding operator\. Following the unified formulation in\(Liet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib9)\), abb\-bit quantized value can be written as

Q​\(X;b,e,α\)=α⋅\(−1\)sign⋅2d⋅\(1\+∑i=1b−1−emi2i\),Q\(X;\\,b,\\,e,\\,\\alpha\)=\\alpha\\cdot\(\-1\)^\{\\mathrm\{sign\}\}\\cdot 2^\{d\}\\cdot\\Bigl\(1\+\\sum\_\{i=1\}^\{b\-1\-e\}\\frac\{m\_\{i\}\}\{2^\{i\}\}\\Bigr\),\(4\)wheresign∈\{0,1\}\\mathrm\{sign\}\\in\\\{0,1\\\}, the exponentdduseseebits, each mantissa bitmi∈\{0,1\}m\_\{i\}\\in\\\{0,1\\\}, and the scaling factorα\\alphais determined by the dynamic range of a quantization group \(per\-tensor, per\-channel, or per\-block\)\. In practice, inference stacks apply Eq\. \([4](https://arxiv.org/html/2607.15810#S2.E4)\) independently to weights and activations before GEMM; we denote abb\-bit weight–activation configuration as WbbAb′b^\{\\prime\}when both operands usebbandb′b^\{\\prime\}bits, respectively\.

Floating\-point quantization keeps a non\-zero exponent field \(e\>0e\>0in Eq\. \([4](https://arxiv.org/html/2607.15810#S2.E4)\)\), yielding a non\-uniform grid that concentrates representable levels near zero and preserves wider dynamic range at the cost of irregular spacing between levels\.

##### FP8\.

The E4M3 format used in LLM inference setsb=8b=8,e=4e=4, and three mantissa bits, providing 256 positive representable levels per sign and a maximum representable magnitude ofΔmax=448\\Delta\_\{\\max\}=448\. FP8 W8A8 GEMMs are natively supported on Hopper and Blackwell FP8 Tensor Cores and underpin FP8 rollout in recent RL systems\(Xiet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib11); Qiuet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib12); Liet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib9)\)\. DeepGEMM\(Zhaoet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib14)\)implements high\-throughput FP8 GEMMs with*fine\-grained*block scaling: rather than a single scale per tensor, each tile of values carries its ownα\\alphain Eq\. \([4](https://arxiv.org/html/2607.15810#S2.E4)\), matching the block\-wise layouts of large MoE models more closely than per\-tensor schemes\. The library JIT\-compiles kernels at runtime \(fp8\_gemm\_nt,m\_grouped\_fp8\_gemm\_\*\) and is widely used for FP8 inference prefilling, making FP8 rollout a mature acceleration path for RL\.

##### FP4\.

Where FP8 E4M3 already trades a 256\-level grid for roughly2×2\\timesGEMM throughput over BF16, FP4 pushes the same trade\-off one step further: halving bit width again targets roughly4×4\\timesthroughput, but shrinks the representable set to only eight positive magnitudes per sign under a typical E2M1 core \(b=4b=4,e=2e=2, one mantissa bit\)—32×32\\timescoarser than E4M3\. The resulting quantization error is larger and less uniformly distributed, so FP4 rollout amplifies the training–inference mismatch of Section[2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2)beyond what FP8 IS corrections typically absorb\. NVFP4 is NVIDIA’s block\-scaled instantiation of this precision tier, native to Blackwell FP4 Tensor Cores\(Alvarezet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib1); NVIDIA,[2025](https://arxiv.org/html/2607.15810#bib.bib2); Jarmusch and Chandrasekaran,[2025](https://arxiv.org/html/2607.15810#bib.bib3)\)\. In Eq\. \([4](https://arxiv.org/html/2607.15810#S2.E4)\), the NVFP4*core*uses E2M1 encoding and yields\{0,0\.5,1,1\.5,2,3,4,6\}\\\{0,0\.5,1,1\.5,2,3,4,6\\\}per sign; sixteen consecutive elements share an E4M3*block scale*that sets their commonα\\alpha, and an optional per\-tensor FP32 scale provides a second level of dynamic\-range control\(Alvarezet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib1)\)\.

## 3Methods

Naive NVFP4 W4A4 rollout with BF16 training fails because low\-precision inference induces a large training–inference mismatch that destabilizes optimization\. This section develops a dual\-side alignment strategy\. Section[3\.1](https://arxiv.org/html/2607.15810#S3.SS1)first models the training–inference forward error and shows experimentally that activation FP4 quantization is the dominant and harder\-to\-align source of the log\-probability gap between rollout and learner\. Section[3\.2](https://arxiv.org/html/2607.15810#S3.SS2)then derives asymmetric W4A16 QAT: for trainer\-side, we fake\-quantize weights to FP4, but deliberately keep training activations in BF16\. Section[3\.3](https://arxiv.org/html/2607.15810#S3.SS3)finally reduces the remaining rollout\-side activation error through targeted residual compensation\.

Table 1:Training\-side QDQ has opposite effects on weight and activation mismatch\. Weight QDQ aligns W4A16 rollout with the learner, whereas activation QDQ does not shrink W16A4 mismatch because the two engines quantize already\-different activations\.### 3\.1FP4 RL Mismatch Error Analysis

Figure[1](https://arxiv.org/html/2607.15810#S1.F1)shows the symptom: W4A4 rollout with a BF16 learner collapses within roughly 150 optimization steps, while BF16 rollout remains stable\. For a fixed checkpoint and token sequence, letπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}be the rollout policy evaluated by the inference engine andπθ\\pi\_\{\\theta\}be the learner policy evaluated by the training engine\. We write the per\-token gapδt\\delta\_\{t\}at a coarse level as

δt≈δtweight\+δtact\+δtnum,\\delta\_\{t\}\\;\\approx\\;\\delta\_\{t\}^\{\\mathrm\{weight\}\}\+\\delta\_\{t\}^\{\\mathrm\{act\}\}\+\\delta\_\{t\}^\{\\mathrm\{num\}\},\(5\)where the three terms correspond to weight quantization, activation quantization, and numerical differences between decoupled engines\(Qiet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib6); Yaoet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib7)\)\. The key question is which term can be reduced by standard quantization–dequantization \(QDQ\) alignment, and which term remains exposed in true W4A4 rollout\.

##### Layer\-wise source of the mismatch\.

Consider one linear or MoE projection and let the BF16 training output be

𝐘train=𝐖train​𝐗train\.\\mathbf\{Y\}\_\{\\mathrm\{train\}\}=\\mathbf\{W\}\_\{\\mathrm\{train\}\}\\mathbf\{X\}\_\{\\mathrm\{train\}\}\.\(6\)The inference engine receives synchronized weights but recomputes activations through a different numerical path before applying NVFP4 quantization\. We denote the weight quantization error by

Δ​𝐖q=QW​\(𝐖infer\)−𝐖infer,\\Delta\\mathbf\{W\}\_\{q\}=Q\_\{W\}\(\\mathbf\{W\}\_\{\\mathrm\{infer\}\}\)\-\\mathbf\{W\}\_\{\\mathrm\{infer\}\},\(7\)whereQWQ\_\{W\}is the hardware\-aligned NVFP4 weight QDQ operator\. For activations, the inference\-side BF16 activation differs from the learner activation even before FP4 quantization:

𝜼=𝐗infer−𝐗train\.\\boldsymbol\{\\eta\}=\\mathbf\{X\}\_\{\\mathrm\{infer\}\}\-\\mathbf\{X\}\_\{\\mathrm\{train\}\}\.\(8\)After activation quantization,

Δ​𝐗q=QA​\(𝐗infer\)−𝐗infer,\\Delta\\mathbf\{X\}\_\{q\}=Q\_\{A\}\(\\mathbf\{X\}\_\{\\mathrm\{infer\}\}\)\-\\mathbf\{X\}\_\{\\mathrm\{infer\}\},\(9\)whereQAQ\_\{A\}is the NVFP4 activation QDQ operator\. The inference output can therefore be written as

𝐘infer=\(𝐖train\+Δ​𝐖q\)​\(𝐗train\+𝜼\+Δ​𝐗q\)\.\\mathbf\{Y\}\_\{\\mathrm\{infer\}\}=\(\\mathbf\{W\}\_\{\\mathrm\{train\}\}\+\\Delta\\mathbf\{W\}\_\{q\}\)\(\\mathbf\{X\}\_\{\\mathrm\{train\}\}\+\\boldsymbol\{\\eta\}\+\\Delta\\mathbf\{X\}\_\{q\}\)\.\(10\)Subtracting Eq\. \([6](https://arxiv.org/html/2607.15810#S3.E6)\) and taking the Frobenius norm defines the layer\-wise forward mismatch

Δ​𝐘≜‖𝐘infer−𝐘train‖2=‖𝐖train⋆\(𝜼\+Δ​𝐗q\)⏟activation​error\+Δ​𝐖q⏟weight​error⋆𝐗train\+Δ​𝐖q​\(𝜼\+Δ​𝐗q\)⏟second​order‖2\.\\Delta\\mathbf\{Y\}\\triangleq\\left\\\|\\mathbf\{Y\}\_\{\\mathrm\{infer\}\}\-\\mathbf\{Y\}\_\{\\mathrm\{train\}\}\\right\\\|\_\{2\}=\\left\\\|\\mathbf\{W\}\_\{\\mathrm\{train\}\}\\star\\underbrace\{\(\\boldsymbol\{\\eta\}\+\\Delta\\mathbf\{X\}\_\{q\}\)\}\_\{\\mathrm\{activation\\ error\}\}\+\\underbrace\{\\Delta\\mathbf\{W\}\_\{q\}\}\_\{\\mathrm\{weight\\ error\}\}\\star\\mathbf\{X\}\_\{\\mathrm\{train\}\}\+\\underbrace\{\\Delta\\mathbf\{W\}\_\{q\}\(\\boldsymbol\{\\eta\}\+\\Delta\\mathbf\{X\}\_\{q\}\)\}\_\{\\mathrm\{second\\ order\}\}\\right\\\|\_\{2\}\.\(11\)Equation \([11](https://arxiv.org/html/2607.15810#S3.E11)\) is the central distinction\. The weight error is solely a quantization error, while the activation error contains not only the FP4 quantization errorΔ​𝐗q\\Delta\\mathbf\{X\}\_\{q\}, but also the engine mismatch𝜼\\boldsymbol\{\\eta\}; Since the inference side weights are replicated from the training side, ensuring𝐖infer=𝐖train\\mathbf\{W\}\_\{\\mathrm\{infer\}\}=\\mathbf\{W\}\_\{\\mathrm\{train\}\}, applying the same quantization function to the weights on the training side is sufficient to eliminate the weight error, i\.e\.,‖Δ​𝐖q‖2=0\\\|\\Delta\\mathbf\{W\}\_\{q\}\\\|\_\{2\}=0\. However, activation error is hard to align\. In Figure[2](https://arxiv.org/html/2607.15810#S3.F2), we conduct two experiments, namely Weight QDQ Alignment and Activation QDQ Alignment, which illustrate QDQ alignment only works for weights, but not for activations\.

![Refer to caption](https://arxiv.org/html/2607.15810v1/x4.png)

\(a\) Weight QDQ Alignment

![Refer to caption](https://arxiv.org/html/2607.15810v1/x5.png)

\(b\) Activation QDQ Alignment

Figure 2:Per\-token log\-probability gap under the motivating ablation\. W16A4 is much closer to W4A4 than to W4A16, identifying activation quantization as the dominant source of mismatch\. Weight QDQ reduces the W4A16 gap while activation QDQ does not reduce the W16A4 gap\.

### 3\.2Asymmetric Quantization\-Aware Training for Training\-side

Based on the above experimental observations, in the setting of FP4 W4A4 RL, although rollout uses W4A4, we propose an asymmetric QAT on the training\-side, that is, perform FP4 QAT on the weights on the trainer, and maintain BF16 for the activations\.

##### Forward and backward of Asymmetric QAT\.

For each linear or MoE projection, letQDQW​\(⋅\)\\mathrm\{QDQ\}\_\{W\}\(\\cdot\)denote the FP4 weight fake\-quantization operator whose scale layout and rounding match the rollout kernel\. The asymmetric QAT forward pass is

𝐖^=QDQW​\(𝐖\),𝐘qat=𝐖^​𝐗trainBF16,\\widehat\{\\mathbf\{W\}\}=\\mathrm\{QDQ\}\_\{W\}\(\\mathbf\{W\}\),\\qquad\\mathbf\{Y\}\_\{\\mathrm\{qat\}\}=\\widehat\{\\mathbf\{W\}\}\\mathbf\{X\}\_\{\\mathrm\{train\}\}^\{\\mathrm\{BF16\}\},\(12\)with no activation fake quantization on the learner\. The backward pass uses the straight\-through estimator \(STE\) for the QDQ node:

∂ℒ∂𝐖≈∂ℒ∂𝐖^,∂ℒ∂𝐗train=𝐖^⊤​∂ℒ∂𝐘qat\.\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\mathbf\{W\}\}\\approx\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\widehat\{\\mathbf\{W\}\}\},\\qquad\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\mathbf\{X\}\_\{\\mathrm\{train\}\}\}=\\widehat\{\\mathbf\{W\}\}^\{\\top\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\mathbf\{Y\}\_\{\\mathrm\{qat\}\}\}\.\(13\)Thus the optimizer still updates BF16 master weights, but every learner forward pass sees the same discretized weight values used by rollout\.

To demonstrate why trainer\-side W4A16 QAT is superior, we conduct the following analysis\. To make the contrast explicit, letΔ​𝐘pre\\Delta\\mathbf\{Y\}^\{\\mathrm\{pre\}\}andΔ​𝐘post\\Delta\\mathbf\{Y\}^\{\\mathrm\{post\}\}denote the layer\-wise mismatch norm before and after inserting the corresponding QDQ node into the learner forward pass\.

##### Why weights should be quantized on the learner\.

We use the weight\-only quantization setting to demonstrate the effect of weight error\. In this setting, activations remain BF16 and only the weight term in Eq\. \([11](https://arxiv.org/html/2607.15810#S3.E11)\) is active, then the mismatch is

Δ​𝐘pre=‖𝐖train⋆𝜼\+Δ​𝐖q⋆𝐗train\+Δ​𝐖q​𝜼‖2\.\\Delta\\mathbf\{Y\}^\{\\mathrm\{pre\}\}=\\left\\\|\\mathbf\{W\}\_\{\\mathrm\{train\}\}\\star\\boldsymbol\{\\eta\}\+\\Delta\\mathbf\{W\}\_\{q\}\\star\\mathbf\{X\}\_\{\\mathrm\{train\}\}\+\\Delta\\mathbf\{W\}\_\{q\}\\boldsymbol\{\\eta\}\\right\\\|\_\{2\}\.\(14\)After utilizing weight fake\-quantization on the learner, both sides multiply by the same quantized weight\. This is because the rollout engines load a bit\-exact copy of the trainer parameters at each step:

𝐖infer=𝐖train\.\\mathbf\{W\}\_\{\\mathrm\{infer\}\}=\\mathbf\{W\}\_\{\\mathrm\{train\}\}\.\(15\)If the training\-side quantizationQW′Q\_\{W\}^\{\{\}^\{\\prime\}\}matches the rollout\-side, then the weight quantization error vanishes:

‖Δ​𝐖q‖2=‖QW​\(𝐖infer\)−QW′​\(𝐖train\)‖2=0\.\\left\\\|\\Delta\\mathbf\{W\}\_\{q\}\\right\\\|\_\{2\}=\\left\\\|Q\_\{W\}\(\\mathbf\{W\}\_\{\\mathrm\{infer\}\}\)\-Q\_\{W\}^\{\{\}^\{\\prime\}\}\(\\mathbf\{W\}\_\{\\mathrm\{train\}\}\)\\right\\\|\_\{2\}=0\.\(16\)
So the train–inference mismatch becomesΔ​𝐘post=‖𝐖train⋆𝜼‖2\\Delta\\mathbf\{Y\}^\{\\mathrm\{post\}\}=\\left\\\|\\mathbf\{W\}\_\{\\mathrm\{train\}\}\\star\\boldsymbol\{\\eta\}\\right\\\|\_\{2\}\. SinceΔ​𝐖q​𝐗train\\Delta\\mathbf\{W\}\_\{q\}\\mathbf\{X\}\_\{\\mathrm\{train\}\}is generally nonzero, then weight QDQ is guaranteed to shrink the mismatch:

Δ​𝐘post<Δ​𝐘pre\.\\Delta\\mathbf\{Y\}^\{\\mathrm\{post\}\}<\\Delta\\mathbf\{Y\}^\{\\mathrm\{pre\}\}\.\(17\)
![Refer to caption](https://arxiv.org/html/2607.15810v1/x6.png)

\(a\) Training Reward

![Refer to caption](https://arxiv.org/html/2607.15810v1/x7.png)

\(b\) AIME24 Test Score

Figure 3:W4A16 QAT improves over naive W4A4 but does not match BF16 RL stability\.\(a\)Mean training reward: W4A16 QAT with W4A4 inference \(red\) avoids collapse yet grows more slowly than the BF16 rollout baseline \(blue\)\.\(b\)AIME24 test score shows the same gap: red remains below blue during almost all the training\.Why activation should remain unquantized on the learner\.We use the activation\-only quantization setting to demonstrate the effect of activation error\. In this setting, weights remain BF16 and only the activation term in Eq\. \([11](https://arxiv.org/html/2607.15810#S3.E11)\) is active, then the mismatch is

Δ​𝐘pre=‖𝐖train​\(𝜼\+Δ​𝐗q\)‖2\.\\Delta\\mathbf\{Y\}^\{\\mathrm\{pre\}\}=\\left\\\|\\mathbf\{W\}\_\{\\mathrm\{train\}\}\(\\boldsymbol\{\\eta\}\+\\Delta\\mathbf\{X\}\_\{q\}\)\\right\\\|\_\{2\}\.\(18\)After inserting activation QDQ on the learner, the comparison becomes

Δ​𝐘post=‖𝐖train​\[QA​\(𝐗train\+𝜼\)−QA​\(𝐗train\)\]‖2,\\Delta\\mathbf\{Y\}^\{\\mathrm\{post\}\}=\\left\\\|\\mathbf\{W\}\_\{\\mathrm\{train\}\}\\bigl\[Q\_\{A\}\(\\mathbf\{X\}\_\{\\mathrm\{train\}\}\+\\boldsymbol\{\\eta\}\)\-Q\_\{A\}\(\\mathbf\{X\}\_\{\\mathrm\{train\}\}\)\\bigr\]\\right\\\|\_\{2\},\(19\)because the two engines now applyQAQ\_\{A\}to already\-different activations\. Unlike the weight case, applying activation QDQ does not guarantee to reduce and can even widen the training–inference gap\. This situation is verified in Figure[7](https://arxiv.org/html/2607.15810#S4.F7), the activation QDQ does not reduce the training–inference gap, and even increases it\.

### 3\.3Activation Error Compensation for Rollout\-side

Although W4A16 QAT on the training\-side can reduce some train\-inference mismatch to avoid the catastrophic collapse of naive FP4 RL, it is still not enough to match the BF16 baseline\. As shown in Figure[3](https://arxiv.org/html/2607.15810#S3.F3), W4A16 QAT with W4A4 rollout avoids the catastrophic collapse, but its reward grows more slowly and its evaluation score remains below the BF16 rollout baseline\.

The remaining bottleneck is the activation part of Eq\. \([11](https://arxiv.org/html/2607.15810#S3.E11)\)\. Only considering the training side cannot reduce the activation error within it\. To minimize the activation error, we continue from the inference side and consider the activation quantization errorΔ​𝐗q\\Delta\\mathbf\{X\}\_\{q\}of it\.

For a BF16 activation𝐗\\mathbf\{X\}, the definition of the activation quantization error is as follows:

Δ​𝐗q=QA​\(𝐗\)−𝐗,\\Delta\\mathbf\{X\}\_\{q\}=Q\_\{A\}\(\\mathbf\{X\}\)\-\\mathbf\{X\},\(20\)which measures the error introduced by FP4 activation quantization\. This quantization error is not uniform across hidden dimensions\. Because block scales are determined by local dynamic range, some high\-magnitude channels create large quantization errors, while other channels have much smaller quantization errors\. Figure[4](https://arxiv.org/html/2607.15810#S3.F4)shows this per\-channel structure: high\-residual channels appear as persistent vertical bands\. This motivates allocating additional computation only to channels with large‖Δ​𝐗q‖\\\|\\Delta\\mathbf\{X\}\_\{q\}\\\|\.

##### Targeted residual compensation\.

Rather than usingQA​\(𝐗\)Q\_\{A\}\(\\mathbf\{X\}\)directly, we correct the quantization error on a selected channel set𝒮\\mathcal\{S\}:

𝐗~=QA​\(𝐗\)−𝐌𝒮⊙QA​\(Δ​𝐗q\),\\tilde\{\\mathbf\{X\}\}\\;=\\;Q\_\{A\}\(\\mathbf\{X\}\)\-\\mathbf\{M\}\_\{\\mathcal\{S\}\}\\odot Q\_\{A\}\(\\Delta\\mathbf\{X\}\_\{q\}\),\(21\)where𝐌𝒮\\mathbf\{M\}\_\{\\mathcal\{S\}\}is a binary channel mask and⊙\\odotdenotes element\-wise multiplication\. Channels outside𝒮\\mathcal\{S\}follow the standard W4A4 path\. Channels inside𝒮\\mathcal\{S\}receive a second FP4 correction term, reducing the distance to the BF16 reference:

‖𝐗~−𝐗train‖2<‖QA​\(𝐗\)−𝐗train‖2,\\bigl\\\|\\tilde\{\\mathbf\{X\}\}\-\\mathbf\{X\}\_\{\\mathrm\{train\}\}\\bigr\\\|\_\{2\}\\;<\\;\\bigl\\\|Q\_\{A\}\(\\mathbf\{X\}\)\-\\mathbf\{X\}\_\{\\mathrm\{train\}\}\\bigr\\\|\_\{2\},\(22\)thereby shrinking the activation contribution to the log\-probability gap from the inference side\. The correction is residual\-based rather than full BF16 fallback, so the main GEMM remains on the W4A4 FP4 Tensor Core path\.

##### Online per\-channel selection\.

RL changes the policy every update, so activation statistics are non\-stationary\. We therefore choose𝒮\\mathcal\{S\}online for each forward pass\. For channeljj, define the residual score

sj=‖Δ​𝐗q​\[:,j\]‖2,s\_\{j\}=\\left\\\|\\Delta\\mathbf\{X\}\_\{q\}\[:,j\]\\right\\\|\_\{2\},\(23\)aggregated over the current token batch, and keep the largestk%k\\%channels:

𝒮=TopKk%⁡\(\{sj\}j=1d\)\.\\mathcal\{S\}=\\operatorname\{TopK\}\_\{k\\%\}\\left\(\\\{s\_\{j\}\\\}\_\{j=1\}^\{d\}\\right\)\.\(24\)In our implementation we setk=50k=50for the profiled W13 activation site, which captures the large residual channels while adding only about∼\\sim10% additional FLOPs relative to plain W4A4 rollout\. With W4A16 QAT plus this rollout\-side compensation, the resulting training curve closely matches BF16 rollout and training\. Figure[4](https://arxiv.org/html/2607.15810#S3.F4)\(b\) confirms this at the tensor level: the per\-layer MoE output mismatch under W4A16 QAT with residual compensation \(orange\) approaches the BF16 baseline \(blue\), whereas W4A16 QAT alone \(purple\) or no QAT \(green\) leaves a substantially larger gap\.

##### Efficient implementation\.

A naive implementation of Eqs\. \([20](https://arxiv.org/html/2607.15810#S3.E20)\)–\([21](https://arxiv.org/html/2607.15810#S3.E21)\) launches separate kernels for activation quantization, residual extraction, channel selection, residual quantization, masking, and reconstruction, which would erase much of the W4A4 speed benefit\. We fuse the dominant stages into a Triton kernel, reducing launch overhead and keeping the end\-to\-end cost near the∼\\sim10% FLOP increase from the selected residual channels\. Together, asymmetric W4A16 QAT and inference\-side residual compensation form the intended dual\-side alignment: the learner removes weight quantization mismatch, while the rollout engine directly reduces the activation quantization error that cannot be fixed by training\-side QDQ alone\.

![Refer to caption](https://arxiv.org/html/2607.15810v1/x8.png)\(a\)NVFP4 activation quantization residual motivates top\-k% channel selection\.
![Refer to caption](https://arxiv.org/html/2607.15810v1/x9.png)\(b\)Layer\-wise MoE\-output mismatch under four training–inference configurations\.

Figure 4:\(a\)Absolute NVFP4 activation quantization residual\|Δ​𝐗q\|=\|QA​\(𝐗\)−𝐗\|\|\\Delta\\mathbf\{X\}\_\{q\}\|=\|Q\_\{A\}\(\\mathbf\{X\}\)\-\\mathbf\{X\}\|on a representative MoE activation \(color scale clipped at the 99\.5th percentile for visibility\)\. Residual magnitude varies sharply across hidden dimensions: most channels remain near zero, while a small set of outlier channels form persistent vertical bands across sequence positions, motivating top\-kk% channel selection for targeted compensation\.\(b\)Layer\-wise MoE outputL2L\_\{2\}distance under four training–inference configurations\. Without QAT \(green, W16A16 train / W4A4 infer\), the output mismatch is largest\. W4A16 QAT \(purple\) reduces the gap by aligning weights to the quantized grid\. Adding residual compensation \(orange, W4A4\+Res infer\) further closes the distance toward the BF16 baseline \(blue\), confirming that both weight\-side QAT and activation\-side compensation contribute to reducing the training–inference mismatch progressively across all layers\.

## 4Experiments

We evaluate the proposed dual\-side alignment framework on full\-parameter GRPO training of a large\-scale MoE model under NVFP4 W4A4 rollout\. We first describe the experimental setup \(Section[4\.1](https://arxiv.org/html/2607.15810#S4.SS1)\), followed by the main RL training results \(Section[4\.2](https://arxiv.org/html/2607.15810#S4.SS2)\), ablation studies \(Section[4\.3](https://arxiv.org/html/2607.15810#S4.SS3)\), efficiency analysis \(Section[4\.4](https://arxiv.org/html/2607.15810#S4.SS4)\), and activation mismatch visualization \(Section[4\.5](https://arxiv.org/html/2607.15810#S4.SS5)\)\.

### 4\.1Experimental Setup

##### Model Configuration\.

We conduct experiments on MoE 25B\-A2\.8B model\. Training and rollout are performed on a cluster of NVIDIA Blackwell GPUs\. We adopt a disaggregated architecture that decouples training and inference onto separate node pools\. The training backend uses Megatron\-LM with tensor\-model parallelismTP=2\\mathrm\{TP\}\{=\}2, expert\-model parallelismEP=16\\mathrm\{EP\}\{=\}16, context parallelismCP=1\\mathrm\{CP\}\{=\}1, pipeline parallelismPP=1\\mathrm\{PP\}\{=\}1, and sequence parallelismSP=1\\mathrm\{SP\}\{=\}1, yielding a data\-parallel degree ofDP=1\\mathrm\{DP\}\{=\}1\. The rollout engine is SGLang withtrtllm\_mhaattention backend,TP=2\\mathrm\{TP\}\{=\}2, andDP=1\\mathrm\{DP\}\{=\}1\. Weights are synchronized from the trainer to the rollout engine at each checkpoint refresh\.

##### RL Configuration\.

We train with Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.15810#bib.bib5)\)as follows: Each prompt is sampled withN=16N\{=\}16responses; the global batch size is 256 and the mini\-batch size is 128\. We use a single PPO epoch per update with a learning rate of1​e−61\\mathrm\{e\-\}6and AdamW optimizer \(β1=0\.9\\beta\_\{1\}\{=\}0\.9,β2=0\.95\\beta\_\{2\}\{=\}0\.95,ϵ=1​e−15\\epsilon\{=\}1\\mathrm\{e\-\}15\)\. The maximum prompt length is 3,072 tokens and the maximum generation length is 32,768 tokens\. We adopt asymmetric clipping with positive clip ratio0\.270\.27and negative clip ratio0\.20\.2, a dynamic PG loss strategy with cumulative clip range of 90, and sample\-averaged loss\. Gradient checkpointing is enabled with MoE\-layer recomputation\. Optimizer states and main weights are offloaded to CPU memory to fit the model within the GPU memory budget\. All experiments run for 260 training steps and evaluation every 10 steps\.

##### Datasets and evaluation\.

Training data consists of a mixture of mathematical reasoning and code generation prompts: \(i\) a curated math dataset combining O4\-Mini, verified prover data, and DeepMath with Chinese\-language entries filtered out; \(ii\) a filtered code dataset spanning diverse programming tasks\. We evaluate on two held\-out benchmarks: \(i\)AIME 2024/25 \+ HMMT 2025, a suite of competition\-level mathematics problems; \(ii\)LiveCodeBench\(LCB\), a continuously updated code generation benchmark with Python\-only submissions evaluated via a unified evaluation service\. All evaluation results are reported as pass@1 accuracy averaged over multiple copies\.

Table 2:Pass@1 accuracy \(%\) on four benchmarks\.BF16 RLis the full\-precision baseline;FP8 RLapplies FP8 quantization on both sides;NVFP4 \+ BF16 Trainuses NVFP4 rollout with BF16 training and no alignment;QUADSis our proposed dual\-side alignment pipeline in NVFP4 RL\. Our method achieves accuracy on par with BF16 across all benchmarks, while the naive NVFP4 RL suffers severe degradation\. For each benchmark, the best result is shown in bold and the second\-best result is underlined\.

### 4\.2Main Experiment

We compare the following configurations, for each configuration, we train for 260 steps and report pass@1 accuracy on four held\-out benchmarks: AIME 2024, AIME 2025, HMMT 2025, and LiveCodeBench\.

- •BF16 RL: BF16 rollout and BF16 training\. This is the standard full\-precision baseline with no quantization on either side\.
- •FP8 RL: FP8 rollout and FP8 training\. Both weights and activations are quantized to FP8 at inference time and the trainer applies FP8 training\.
- •Naive NVFP4 RL: NVFP4 W4A4 rollout and BF16 training\. Both weights and activations are quantized to E2M1 at inference time while the trainer operates in BF16\.
- •QUADS\(ours\): The dual\-side alignment pipeline combining Asymmetric QAT \(Section[3\.2](https://arxiv.org/html/2607.15810#S3.SS2)\) with inference\-side residual activation compensation \(Section[3\.3](https://arxiv.org/html/2607.15810#S3.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2607.15810v1/x10.png)

\(a\) LiveCodeBench

![Refer to caption](https://arxiv.org/html/2607.15810v1/x11.png)

\(b\) HMMT 2025

![Refer to caption](https://arxiv.org/html/2607.15810v1/x12.png)

\(c\) AIME 2024

![Refer to caption](https://arxiv.org/html/2607.15810v1/x13.png)

\(d\) AIME 2025

Figure 5:Pass@1 score curves on four held\-out benchmarks\.BF16\(blue\) andFP8 RL\(red\) provide strong baselines\.QUADS\(green, ours\) tracks both baselines closely on every benchmark\.NVFP4 \+ BF16 Train\(orange, naive W4A4\) has lower initial point and collapses after roughly 150 steps\.##### Key observations\.

As illustrated in Figure[5](https://arxiv.org/html/2607.15810#S4.F5)and Table[2](https://arxiv.org/html/2607.15810#S4.T2), we summarize the following:

- •BF16serves as the upper\-bound reference, exhibiting steady score growth across all four benchmarks\.
- •FP8 RLis the standard low\-precision baseline; with FP8 training and FP8 rollout, its learning curve closely tracks BF16 RL\.
- •NVFP4 \+ BF16 Train\(no alignment\) collapses after roughly 150 steps on every benchmark due to unmitigated training–inference mismatch \(Figure[5](https://arxiv.org/html/2607.15810#S4.F5), orange curves\), confirming that a naive NVFP4 setup is incompatible with stable RL training\.
- •QUADS\(ours\) recovers accuracy to within the variance of the BF16 baseline across all four benchmarks, demonstrating that training\-side weight alignment combined with inference\-side residual activation compensation is both necessary and sufficient for stable NVFP4 RL\.

A central failure mode of naive NVFP4 RL is the growing discrepancy between rollout and trainer log\-probabilities, which directly biases importance ratios in GRPO\. We therefore track the maximum per\-step log\-probability difference‖log⁡πθinfer−log⁡πθtrain‖max\\\|\\log\\pi\_\{\\theta\}^\{\\mathrm\{infer\}\}\-\\log\\pi\_\{\\theta\}^\{\\mathrm\{train\}\}\\\|\_\{\\max\}across three configurations:Naive NVFP4 RL\(no alignment\),QUADS\(dual\-side alignment\), andBF16 RL\(full\-precision reference\)\. Figure[6](https://arxiv.org/html/2607.15810#S4.F6)reports this metric over 260 training steps\. Naive NVFP4 RL maintains the largest gap \(∼\\sim1\.3\) throughout training, explaining the policy\-gradient instability observed in Figure[5](https://arxiv.org/html/2607.15810#S4.F5)\. QUADS consistently lowers the gap to∼\\sim0\.86, substantially closing the train–inference discrepancy\. The BF16 baseline \(∼\\sim0\.74\) represents the irreducible engine\-drift floor between Megatron\-LM and SGLang, confirming that our dual\-side alignment approaches this lower bound\.

### 4\.3Ablation Studies

#### 4\.3\.1W4A4 QAT vs\. W4A16 QAT

![Refer to caption](https://arxiv.org/html/2607.15810v1/x14.png)
Figure 6:Maximum log\-probability gap‖log⁡πθinfer−log⁡πθtrain‖max\\\|\\log\\pi\_\{\\theta\}^\{\\mathrm\{infer\}\}\-\\log\\pi\_\{\\theta\}^\{\\mathrm\{train\}\}\\\|\_\{\\max\}over training steps\.Naive NVFP4 RL\(red\) sustains the largest gap;QUADS\(green, ours\) substantially reduces it toward theBF16 RL\(purple\) engine\-drift floor\.![Refer to caption](https://arxiv.org/html/2607.15810v1/x15.png)
Figure 7:Layer\-wise MoE\-input mismatch under two QAT configurations on matched requests\.Green: Asymmetric W4A16 QAT\.Red: Symmetric W4A4 QAT\. Symmetric W4A4 QAT shows larger mismatch\.
We ablate symmetric W4A4 QAT, which fake\-quantizes both weights and activations during training, against asymmetric W4A16 QAT, which fake\-quantizes weights only, on matched requests\. Figure[7](https://arxiv.org/html/2607.15810#S4.F7)reports layer\-wise MoE\-input mismatch under two training–inference configurations:

- •Asymmetric W4A16 QAT\(green\):‖𝐈trainBF16−𝐈inferFP4‖2\\\|\\mathbf\{I\}\_\{\\mathrm\{train\}\}^\{\\mathrm\{BF16\}\}\-\\mathbf\{I\}\_\{\\mathrm\{infer\}\}^\{\\mathrm\{FP4\}\}\\\|\_\{2\}, with weight\-aligned training and W4A4 inference\.
- •Symmetric W4A4 QAT\(red\):‖𝐈trainFP4−𝐈inferFP4‖2\\\|\\mathbf\{I\}\_\{\\mathrm\{train\}\}^\{\\mathrm\{FP4\}\}\-\\mathbf\{I\}\_\{\\mathrm\{infer\}\}^\{\\mathrm\{FP4\}\}\\\|\_\{2\}, with both operands fake\-quantized during training and W4A4 inference\.

Symmetric W4A4 QAT \(red\) exhibits the largest layer\-wise gap, confirming the analysis in Section[3\.2](https://arxiv.org/html/2607.15810#S3.SS2): coarse E2M1 activation fake\-quantization amplifies engine drift rather than reducing it\. By contrast, asymmetric W4A16 QAT \(green\) reduces the activation error on each layer, verifying our asymmetric training design\.

#### 4\.3\.2Effect of Dual\-side Alignment

In this section, we evaluate the effect of our dual\-side alignment method by comparing three configurations:

- •Naive NVFP4: NVFP4 rollout and BF16 training, without alignment on either side\.
- •W4A16 QAT Only: NVFP4 rollout and W4A16 QAT, only with training\-side alignment\.
- •QUADStraining\-side weight alignment*and*inference\-side residual compensation, with both components\.

![Refer to caption](https://arxiv.org/html/2607.15810v1/x16.png)

\(a\) Benchmark Scores

![Refer to caption](https://arxiv.org/html/2607.15810v1/x17.png)

\(b\) AIME24 Test Score

Figure 8:Ablation of training\-side W4A16 QAT and inference\-side residual compensation\.\(a\)Final pass@1 scores across held\-out benchmarks: naive NVFP4 RL without alignment degrades severely \(51\.4% average\), W4A16 QAT alone recovers most of the gap \(71\.1%\), and adding residual compensation yields further consistent gains on every benchmark \(72\.9%\)\.\(b\)AIME24 score over training steps: W4A16 QAT only \(red\) grows more slowly and remains below the BF16 baseline \(blue\), whereas W4A16 QAT \+ Residual \(green\) tracks BF16 closely throughout training\.As shown in Figure[8](https://arxiv.org/html/2607.15810#S4.F8), each alignment component contributes an independent gain\. Figure[8](https://arxiv.org/html/2607.15810#S4.F8)\(a\) shows that naive NVFP4 \+ BF16 Train without any alignment suffers severe accuracy degradation across all benchmarks, confirming the failure mode analyzed in Section[3\.1](https://arxiv.org/html/2607.15810#S3.SS1)\. W4A16 QAT alone stabilizes training and recovers most of the accuracy, yet still leaves a noticeable gap to the BF16 baseline\. Adding inference\-side residual compensation on top of W4A16 QAT further closes this gap: the corrected activation tensors more closely approximate the BF16 reference, reducingδtact\\delta\_\{t\}^\{\\mathrm\{act\}\}and improving every benchmark score\. Figure[8](https://arxiv.org/html/2607.15810#S4.F8)\(b\) corroborates this at the trajectory level on AIME24—W4A16 QAT only lags BF16 throughout training, while the full pipeline restores evaluation growth to near\-baseline levels\. These results confirm that both components contribute independently and complementarily to the final accuracy\.

#### 4\.3\.3Channel Selection Ratio

The residual compensation module selects the top\-kk% of channels by residual magnitude for correction \(Section[3\.3](https://arxiv.org/html/2607.15810#S3.SS3)\)\. We study the sensitivity tokkby varying the selection ratio and measuring the maximum post\-correction log\-probability difference‖log⁡πθinfer−log⁡πθtrain‖max\\\|\\log\\pi\_\{\\theta\}^\{\\mathrm\{infer\}\}\-\\log\\pi\_\{\\theta\}^\{\\mathrm\{train\}\}\\\|\_\{\\max\}between the training and inference engines\.

Table 3:Maximum log\-probability difference between training and inference engines under different residual channel selection ratios\.Table[3](https://arxiv.org/html/2607.15810#S4.T3)reports the maximum log\-probability difference across training steps for each selection ratio\. As the residual channel ratio increases from 12\.5% to 75%, the logprob gap decreases, indicating that correcting more channels further aligns the inference engine with the training engine\. However, larger ratios also incur proportionally higher compute overhead, so the optimalkkbalances accuracy recovery against throughput cost\.

### 4\.4Efficiency Analysis

#### 4\.4\.1Residual Compensation FLOPs Cost

The residual compensation module introduces additional computation on top of the standard NVFP4 W4A4 forward pass\. For each transformer layer, the correction involves: \(i\) extracting the top\-kk% residual channels from the activation tensor, \(ii\) re\-quantizing the residual to E2M1, and \(iii\) adding the corrected residual back to the quantized output\. The additional FLOPs are dominated by the element\-wise operations on the selected channels, which scale as𝒪​\(k⋅d⋅s\)\\mathcal\{O\}\(k\\cdot d\\cdot s\)per layer, whereddis the hidden dimension andssis the sequence length\. Withk=50%k\{=\}50\\%, the residual module adds less than 10% FLOPs overhead relative to the base W4A4 GEMM, making it computationally lightweight\.

#### 4\.4\.2Rollout Throughput Comparison

We measure decode throughput \(tokens/s\) on the SGLang engine for three rollout configurations:

- •FP8 W8A8: FP8 rollout as the current production baseline\.
- •NVFP4 W4A4: Plain NVFP4 rollout without residual compensation\.
- •NVFP4 W4A4 \+ Residual\(ours\): NVFP4 rollout with inference\-side activation error compensation\.

![Refer to caption](https://arxiv.org/html/2607.15810v1/x18.png)\(a\)Decode throughput on SGLang across RL training steps\.Naive NVFP4 W4A4\(red\) consistently achieves the highest throughput\.QUADS\(blue, ours\) incurs a modest∼\\sim5–9% slowdown relative to plain W4A4 but remains substantially faster thanFP8\(green\)\.
![Refer to caption](https://arxiv.org/html/2607.15810v1/x19.png)\(b\)Element\-wise activation mismatch\|𝐀trainBF16−𝐀inferBF16\|\|\\mathbf\{A\}\_\{\\mathrm\{train\}\}^\{\\mathrm\{BF16\}\}\-\\mathbf\{A\}\_\{\\mathrm\{infer\}\}^\{\\mathrm\{BF16\}\}\|at Layer 1\. Even without any quantization, structured non\-zero residuals are clearly visible, revealing that kernel\-level implementation differences introduce an irreducible*engine drift*\.

Figure 9:\(a\) Rollout throughput comparison across training steps\. \(b\) BF16 Train–infer activation mismatch heatmap at Layer 27, showing irreducible engine drift between Megatron\-LM and SGLang even in BF16\.Figure[9\(a\)](https://arxiv.org/html/2607.15810#S4.F9.sf1)reports throughput at several steps during RL training\. Plain NVFP4 W4A4 rollout delivers roughly1\.3×1\.3\\timesthe FP8 throughput, reflecting the benefit of FP4 Tensor Core kernels under our production workload\. Adding residual compensation reduces throughput by only∼\\sim5–9% compared with plain W4A4, owing to fused Triton kernels that combine quantization, residual extraction, and masked reconstruction with minimal launch overhead\. The full pipeline therefore retains a clear speed advantage over FP8 throughput by∼\\sim16%, indicating that the accuracy gains from residual compensation come at an acceptable throughput cost\.

### 4\.5Visualization

#### 4\.5\.1Training–Inference Activation Mismatch

To understand the fundamental source of train\-infer mismatch, we visualize the element\-wise activation differences between the BF16 training engine \(Megatron\-LM\) and the BF16 inference engine \(SGLang\) on identical input batches\. Figure[9\(b\)](https://arxiv.org/html/2607.15810#S4.F9.sf2)plots\|𝐀trainBF16−𝐀inferBF16\|\|\\mathbf\{A\}\_\{\\mathrm\{train\}\}^\{\\mathrm\{BF16\}\}\-\\mathbf\{A\}\_\{\\mathrm\{infer\}\}^\{\\mathrm\{BF16\}\}\|as a heatmap at Layer 1, where each pixel corresponds to a single element indexed by sequence position and hidden dimension\. Even without any quantization, the two engines produce non\-identical activations due to differences in kernel implementations, accumulation order, and numerical rounding\. The heatmap reveals that the mismatch is not uniformly distributed: certain channels exhibit consistently larger residuals, suggesting that specific hidden dimensions are more sensitive to implementation\-level numerical discrepancies\. This irreducible*engine drift*constitutes the lower bound of training–inference mismatch: any quantization scheme can at best match this gap, but never eliminate it\. Our dual\-side alignment framework is designed to approach this lower bound by minimizing the additional mismatch introduced by quantization on both sides\.

## 5Related Work

### 5\.1Reinforcement Learning for Large Language Models

Reinforcement learning from human feedback \(RLHF\) and its verifiable\-reward variant \(RLVR\) have become standard post\-training stages for aligning large language models\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.15810#bib.bib4); Shaoet al\.,[2024](https://arxiv.org/html/2607.15810#bib.bib5)\)\. PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.15810#bib.bib4)\)introduced the clipped surrogate objective that bounds policy updates within a trust region, while GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.15810#bib.bib5)\)removes the value model and normalizes advantages within a sampled response group, substantially simplifying the training pipeline for reasoning tasks\. As chain\-of\-thought traces grow longer, autoregressive rollout on a separate inference engine dominates end\-to\-end wall\-clock time, motivating a line of work on accelerating the RL loop through system\-level optimizations\(Yaoet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib7); Liet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib9); Xiet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib11)\)\.

### 5\.2Low\-Precision Inference and RL Rollout

Low\-precision inference has matured from INT8\(Liet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib9)\)through FP8\(Zhaoet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib14); Xiet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib11); Qiuet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib12)\)to the emerging FP4 tier\. FP8 E4M3, natively supported on Hopper and Blackwell Tensor Cores, provides 256 representable levels per sign and has become the default precision for high\-throughput LLM serving\. NVFP4\(Alvarezet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib1); NVIDIA,[2025](https://arxiv.org/html/2607.15810#bib.bib2)\)further halves the bit width to E2M1 with block\-scaled E4M3 scaling, targeting∼\\sim4×\\timesGEMM throughput over BF16 on Blackwell FP4 Tensor Cores\(Jarmusch and Chandrasekaran,[2025](https://arxiv.org/html/2607.15810#bib.bib3)\)\. When such quantized inference is used for RL rollout, quantization\-aware training \(QAT\) can reduce rollout–trainer mismatch by inserting fake quantization–dequantization nodes into the learner\(Guet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib10)\)\. For weight\-only quantization, QAT is relatively straightforward: both engines multiply by the same QDQ\-aligned weight, and the straight\-through estimator propagates gradients through the rounding operator\. QeRL\(Huanget al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib8)\)applies NVFP4 weight\-only quantization to RL but keeps activations in BF16, executing FP16 GEMMs rather than FP4 Tensor Core operations—thus sidestepping activation quantization entirely\. Symmetric W4A4 QAT, which fake\-quantizes both weights and activations, appears to offer full alignment but in practice amplifies the engine drift𝜼\\boldsymbol\{\\eta\}through the coarse E2M1 decision boundaries \(Section[3\.2](https://arxiv.org/html/2607.15810#S3.SS2)\)\. Our work identifies this operand asymmetry and proposes asymmetric W4A16 QAT combined with inference\-side residual compensation, enabling the stable full\-parameter MoE RL with native W4A4 FP4 GEMM on Blackwell hardware\.

### 5\.3Training–Inference Mismatch in RL

When rollout and training run on separate engines at different precisions, the per\-token log\-probability gapδt\\delta\_\{t\}corrupts importance ratios and destabilizes policy\-gradient updates\(Qiet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib6); Yaoet al\.,[2025](https://arxiv.org/html/2607.15810#bib.bib7)\)\. A series of recent works address this mismatch at the FP8 and INT8 precision tiers\. QuRL\(Liet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib9)\)quantizes the actor for generation while keeping full\-precision gradient updates, using adaptive clipping to control importance ratios\. Jet\-RL\(Xiet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib11)\)enforces a unified FP8 precision flow across training and rollout, eliminating the precision gap at the cost of quantizing the learner\. FP8\-RL\(Qiuet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib12)\)provides a practical low\-precision stack with truncated importance sampling, while QaRL\(Guet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib10)\)aligns rollout and training through rollout\-aware quantization\-aware training\. AIS\(Zhouet al\.,[2026](https://arxiv.org/html/2607.15810#bib.bib13)\)introduces adaptive importance sampling that dynamically adjusts clipping thresholds based on observed ratio distributions\. All of these methods operate on the comparatively fine FP8 or INT8 grid, where\|δt\|\|\\delta\_\{t\}\|remains tractable; none targets native W4A4 FP4 Tensor Core inference, where the E2M1 grid is32×32\\timescoarser than E4M3 and importance\-sampling corrections alone prove insufficient\.

## 6Conclusion

We study NVFP4 rollout for Mixture\-of\-Experts \(MoE\) Reinforcement Learning on NVIDIA Blackwell\. Although native W4A4 FP4 GEMMs offer higher rollout throughput than FP8, directly combining NVFP4 rollout with BF16 training causes unstable GRPO training\. Through training–inference error analysis and controlled ablations, we identify activation quantization, rather than weight quantization, as the dominant source of FP4 RL instability\.

Based on this finding, we propose dual\-side alignment: asymmetric W4A16 quantization\-aware training aligns the trainer\-side weight path, while residual activation compensation reduces rollout\-side activation error without leaving native W4A4 GEMMs\. On full\-parameter GRPO for the MoE model, the proposed NVFP4 pipeline reaches BF16\-level accuracy, improves average pass@1 by 21\.49 percentage points over naive NVFP4 RL, and retains a 16% rollout throughput advantage over FP8\. Future work includes more adaptive residual\-channel selection and extending this alignment principle to other RL algorithms and model architectures\.

## References

- E\. Alvarez, O\. Almog, E\. Chung, S\. Layton, D\. Stosic, R\. Krashinsky, and K\. Aubrey \(2025\)Introducing NVFP4 for efficient and accurate low\-precision inference\.Note:NVIDIA Technical BlogExternal Links:[Link](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/)Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p3.4),[§2\.2](https://arxiv.org/html/2607.15810#S2.SS2.SSS0.Px2.p1.7),[§5\.2](https://arxiv.org/html/2607.15810#S5.SS2.p1.3)\.
- H\. Gu, H\. Wang, J\. Liu, L\. Li, Q\. Zhu, B\. Liu, B\. Xu, L\. Wang, X\. Yang, S\. Lin, S\. Han, and Y\. Guo \(2026\)QaRL: rollout\-aligned quantization\-aware RL for fast and stable training under training–inference mismatch\.arXiv preprint arXiv:2604\.07853\.Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p1.8),[§5\.2](https://arxiv.org/html/2607.15810#S5.SS2.p1.3),[§5\.3](https://arxiv.org/html/2607.15810#S5.SS3.p1.3)\.
- W\. Huang, Y\. Ge, S\. Yang, Y\. Xiao, H\. Mao, Y\. Lin, H\. Ye, S\. Liu, K\. C\. Cheung, H\. Yin, Y\. Lu, X\. Qi, S\. Han, and Y\. Chen \(2025\)QeRL: beyond efficiency–quantization\-enhanced reinforcement learning for LLMs\.arXiv preprint arXiv:2510\.11696\.Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p5.1),[§5\.2](https://arxiv.org/html/2607.15810#S5.SS2.p1.3)\.
- A\. Jarmusch and S\. Chandrasekaran \(2025\)Microbenchmarking NVIDIA’s blackwell architecture: an in\-depth architectural analysis\.arXiv preprint arXiv:2512\.02189\.Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p3.4),[§2\.2](https://arxiv.org/html/2607.15810#S2.SS2.SSS0.Px2.p1.7),[§5\.2](https://arxiv.org/html/2607.15810#S5.SS2.p1.3)\.
- Y\. Li, R\. Elangovan, X\. Dong, P\. Panda, and B\. Khailany \(2026\)QuRL: efficient reinforcement learning with quantized rollout\.arXiv preprint arXiv:2602\.13953\.Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p1.1),[§1](https://arxiv.org/html/2607.15810#S1.p2.1),[§1](https://arxiv.org/html/2607.15810#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p2.2),[§2\.2](https://arxiv.org/html/2607.15810#S2.SS2.SSS0.Px1.p1.4),[§2\.2](https://arxiv.org/html/2607.15810#S2.SS2.p1.3),[§5\.1](https://arxiv.org/html/2607.15810#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2607.15810#S5.SS2.p1.3),[§5\.3](https://arxiv.org/html/2607.15810#S5.SS3.p1.3)\.
- NVIDIA \(2025\)Pretraining large language models with NVFP4\.arXiv preprint arXiv:2509\.25149\.Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p3.4),[§2\.2](https://arxiv.org/html/2607.15810#S2.SS2.SSS0.Px2.p1.7),[§5\.2](https://arxiv.org/html/2607.15810#S5.SS2.p1.3)\.
- P\. Qi, Z\. Liu, X\. Zhou, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025\)Defeating the training\-inference mismatch via FP16\.arXiv preprint arXiv:2510\.26788\.Cited by:[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p1.8),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p3.3),[§3\.1](https://arxiv.org/html/2607.15810#S3.SS1.p1.4),[§5\.3](https://arxiv.org/html/2607.15810#S5.SS3.p1.3)\.
- Z\. Qiu, S\. Yu, J\. Zhang, S\. Zhang, X\. Huang, J\. Yang, and J\. Lai \(2026\)FP8\-RL: a practical and stable low\-precision stack for LLM reinforcement learning\.arXiv preprint arXiv:2601\.18150\.Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p1.1),[§1](https://arxiv.org/html/2607.15810#S1.p2.1),[§1](https://arxiv.org/html/2607.15810#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p1.8),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p2.2),[§2\.2](https://arxiv.org/html/2607.15810#S2.SS2.SSS0.Px1.p1.4),[§5\.2](https://arxiv.org/html/2607.15810#S5.SS2.p1.3),[§5\.3](https://arxiv.org/html/2607.15810#S5.SS3.p1.3)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.InarXiv preprint arXiv:1707\.06347,Cited by:[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px1.p1.4),[§5\.1](https://arxiv.org/html/2607.15810#S5.SS1.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, Y\. K\. Ming, Y\. W\. Zhang, D\. Guo,et al\.\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px1.p1.4),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.p1.2),[§4\.1](https://arxiv.org/html/2607.15810#S4.SS1.SSS0.Px2.p1.7),[§5\.1](https://arxiv.org/html/2607.15810#S5.SS1.p1.1)\.
- H\. Xi, C\. Ruan, P\. Liao, Y\. Lin, H\. Cai, Y\. Zhao, S\. Yang, K\. Keutzer, S\. Han, and L\. Zhu \(2026\)Jet\-RL: enabling on\-policy FP8 reinforcement learning with unified training and rollout precision flow\.arXiv preprint arXiv:2601\.14243\.Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p1.1),[§1](https://arxiv.org/html/2607.15810#S1.p2.1),[§1](https://arxiv.org/html/2607.15810#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p2.2),[§2\.2](https://arxiv.org/html/2607.15810#S2.SS2.SSS0.Px1.p1.4),[§5\.1](https://arxiv.org/html/2607.15810#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2607.15810#S5.SS2.p1.3),[§5\.3](https://arxiv.org/html/2607.15810#S5.SS3.p1.3)\.
- F\. Yao, L\. Liu, D\. Zhang, C\. Dong, J\. Shang, and J\. Gao \(2025\)Your efficient RL framework secretly brings you off\-policy RL training\.Note:Notion BlogExternal Links:[Link](https://fengyao.notion.site/off-policy-rl)Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p1.4),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p2.2),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p3.3),[§3\.1](https://arxiv.org/html/2607.15810#S3.SS1.p1.4),[§5\.1](https://arxiv.org/html/2607.15810#S5.SS1.p1.1),[§5\.3](https://arxiv.org/html/2607.15810#S5.SS3.p1.3)\.
- C\. Zhao, Z\. Xu, L\. Zhao, J\. Li, C\. Xu, A\. Xu, S\. Liu, K\. Zhou, and K\. Yu \(2025\)DeepGEMM: clean and efficient FP8 GEMM kernels with fine\-grained scaling\.Note:GitHubExternal Links:[Link](https://github.com/deepseek-ai/DeepGEMM)Cited by:[§2\.2](https://arxiv.org/html/2607.15810#S2.SS2.SSS0.Px1.p1.4),[§5\.2](https://arxiv.org/html/2607.15810#S5.SS2.p1.3)\.
- J\. Zhou, W\. Shao, L\. Zheng, Y\. Fan, and N\. Wong \(2026\)AIS: adaptive importance sampling for quantized RL\.arXiv preprint arXiv:2605\.13907\.Cited by:[§1](https://arxiv.org/html/2607.15810#S1.p2.1),[§1](https://arxiv.org/html/2607.15810#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.15810#S2.SS1.SSS0.Px2.p2.2),[§5\.3](https://arxiv.org/html/2607.15810#S5.SS3.p1.3)\.