Why do RL runs on LLMs blow up even when the recipe looks right?

GEOALIGN, from the Alibaba team behind Qwen, points at the rollouts. A handful of bad batches push the policy in incoherent directions, and most stability tuning just damps the symptom. This work curates rollouts by their geometry, removing the samples that make update directions conflict before they destabilize training.

Why does it matter?

If instability is largely a bad-batch problem, rollout curation is a lower-effort lever than another round of KL or clip tuning. You fix the data going into the update rather than fighting the optimizer.

Paper: https://arxiv.org/abs/2606.26917

Learn to build effective AI agents in our academy: https://academy.dair.ai

Geometric Rollout Curation for Robust LLM Reinforcement Learning

Source: https://arxiv.org/html/2606.26917

Abstract

Online reinforcement learning is widely used to align large language models (LLMs) with reward signals, yet training can be unstable under noisy or misspecified rewards. We identify a failure mode we calldirectional inconsistency: within a batch, a small set of high-reward rollouts induces representation-space preference directions that sharply disagree with the batch majority, resulting in high-variance and destabilizing updates. We proposeGeoAlign, a lightweight plug-in forrollout curationin iterative policy optimization.GeoAlign(i) forms within-prompt preference pairs, (ii) learns an online projector on per-rollout hidden states to concentrate reward-ordered displacement directions, and (iii) detects directionally inconsistent rollouts via their angular deviation from a batch consensus prototype and rectifies them with within-prompt stable alternatives.GeoAlignis forward-pass only and adds negligible overhead. Across dialogue alignment with a learned reward model and mathematical reasoning with binary verified rewards,GeoAlignimproves final performance and reduces training oscillation, outperforming PF-PPO, PAR, PODS, and Seed-GRPO. These results suggestlatent directional consensusas an effective reliability signal for online LLM RL.

Machine Learning, ICML

1Introduction

Online reinforcement learning (RL) is a standard approach to align large language models (LLMs) with human preferences or task rewards. However,onlineRL fine-tuning is frequently unstable(Casper et al.,2023): learning curves oscillate across iterations, evaluation can plateau early, and policies may regress after apparently beneficial updates. Such instability is most pronounced when rewards are noisy, misspecified, or exploitable(Skalse et al.,2022; Gao et al.,2023)(e.g., reward-model artifacts or reward hacking), where a few high-reward rollouts can dominate the update for the wrong reasons.

Most existing stabilizers treat reward as ascalarreliability signal and only controlhow mucheach rollout contributes to learning (e.g., clipping/shaping(Fu et al.,2025; Yang et al.,2024b), uncertainty-aware weighting(Chen et al.,2025; Li et al.,2025b), or reward-statistics-based filtering(Xu et al.,2025; Zhang et al.,2025b)). Yet policy optimization aggregatesvector-valuedlearning signals: each rollout induces an update direction in a high-dimensional space. Consequently, two rollouts with similarly high rewards can still push the policy toward sharply different directions. When a small fraction of rollouts inducesconflictingupdate directions relative to the batch majority, the stochastic update becomes high-variance and training can oscillate (Fig.1).

Figure 1:Geometric turbulence in preference latent space: within one update step, most preference-induced directions align, while a small fraction forms angular outliers that can destabilize training.We call this failure modedirectional inconsistency: within an RL iteration, a small set of rollouts inducespreference-implied improvement directionsthat are strong angular outliers relative to the batch consensus, despite receiving high rewards. Concretely, we formwithin-promptpreference pairs and map each pair to a latent displacement direction from a lower-reward response to a higher-reward response. Across tasks, these reward-ordered directions exhibit a long-tail geometry: most directions concentrate around a dominant trend, while a small fraction becomes angular outliers (Fig.3). Such outliers are often not low-reward samples; they can be high-reward but unreliable (e.g., reward-model artifacts), and can disproportionately perturb the update.

To address this, we proposeGeoAlign, a lightweight plug-in module forrollout curationin iterative policy optimization. Given a batch of rollouts and their hidden states (detached from the policy),GeoAlignoperates on-the-fly as follows: (i) forms within-prompt preference pairs, (ii) learns a small projector to distill reward-ordered directions into a concentrated manifold, (iii) builds a batch-wise consensus prototype, and (iv) identifies and rectifies only the most directionally inconsistent rollouts using within-prompt stable alternatives.GeoAlignrequiresno per-rollout policy gradientsand adds negligible overhead.

We evaluateGeoAlignacross dialogue alignment (HH-RLHF, continuous reward from ArmoRM) and mathematical reasoning (DAPO-Math-17k, binary verified reward) on Qwen3-1.7B and Qwen3-4B.GeoAlignimproves both final performance and training stability over strong robust-RL baselines (PF-PPO, PAR, PODS, Seed-GRPO), and remains more resilient under controlled reward corruption. Our results suggest latent directional consensus as an effective forward-only reliability signal for robust online LLM RL.

In summary, our contributions are as follows:

• We identifydirectional inconsistency—a geometric failure mode where a few high-reward rollouts imply update directions that sharply conflict with the batch consensus—and empirically characterize its long-tailed angular outliers.

• We proposeGeoAlign, a lightweight, plug-and-play rollout curation module that scores rollouts bylatent directional consensususing only detached hidden states, and conservatively rectifies severe outliers while preserving per-prompt rollout count.

• We show thatGeoAlignimproves both final performance and training stability on dialogue alignment (continuous RM rewards) and mathematical reasoning (binary verified rewards), including controlled reward corruption. Our code is available athttps://github.com/SYSUzhouting/Trinity-RFT.

Conflict of Interest Disclosure:

Authors Daoyuan Chen and Zhenqing Ling are employed by Alibaba Group, the developer of the Qwen3 model series, which are the primary models evaluated in this paper. To ensure transparency, all code and datasets are publicly available, and our results are based strictly on reproducible, quantitative metrics without bias.

2Related Work

Robust Online RL for LLM Alignment.Online RL for aligning LLMs (e.g., GRPO(Shao et al.,2024)) is often brittle under noisy or misspecified rewards. A large body of work improves stability by modifying themagnitudeof the learning signal, such as reward clipping/shaping and conservative update regularization(Fu et al.,2025; Yang et al.,2024b; Jinnai et al.,2025; Cheng et al.,2025a). Another line reweights or filters rollouts/prompts using reward statistics or uncertainty proxies (e.g., semantic entropy), aiming to reduce the influence of unreliable samples(Xu et al.,2025; Zhang et al.,2025b; Chen et al.,2025; Li et al.,2025b). These methods are effective when scalar reward reliability correlates with sample usefulness, but they do not explicitly check whether a rollout’s induced learning signal isdirectionally consistentwith the batch majority, even when rewards are high.

Direction- and Influence-Aware Data Selection.Several recent approaches identify harmful training signals by inspecting per-sample gradients, influence estimates, or trajectory-level attribution(Hu et al.,2025; Dai et al.,2025; Li et al.,2025a; Choe et al.,2024). While conceptually aligned with our goal, these methods typically require additional backward passes, higher-order computation, or parameter-space approximations, making them expensive for online LLM RL with many rollouts per iteration. In contrast,GeoAlignuses only forward-pass hidden states to build a direction-consistency score, enabling lightweight, on-the-fly rollout curation.

Geometric Signals in Representation Learning.Geometric analyses have shown that meaningful supervision can concentrate along structured directions in representation space, studied through alignment/uniformity tradeoffs, manifold structure, and related phenomena(Wang & Isola,2020; Saunshi et al.,2019; Papyan et al.,2020). Compared to self-supervised learning, leveraging such geometric signals for robustness in online RL remains underexplored.GeoAlignbridges this gap by treating within-prompt reward ordering as a source ofpreference directionsand using theirlatent directional consensusas a reliability signal for stabilizing online LLM RL.

3A Geometric Perspective on RL Instability

3.1Preliminaries and Problem Formulation

We consider a standard online RL setting for LLM fine-tuning. At each iteration, for a batch of prompts{xi}\{x_{i}\}, the current policyπθ\pi_{\theta}generatesKKrollouts (responses){yi,k}\{y_{i,k}\}. Each rollout receives a scalar rewardr​(yi,k)r(y_{i,k}), either from a reward model or a verification function. Policy gradient algorithms update the policyθ\thetaby optimizing an objective based on an estimated advantage functionA​(y)A(y). While specific formulations vary, many modern methods(Shao et al.,2024)compute advantage via intra-prompt normalization.

Advantage from Relative Rewards.The advantage for a rolloutyyis often derived from its reward relative to its peers from the same promptxx:

A​(y)∝r​(y)−𝔼y′∼πθ(⋅|x)​[r​(y′)].A(y)\propto r(y)-\mathbb{E}_{y^{\prime}\sim\pi_{\theta}(\cdot|x)}[r(y^{\prime})].(1)This scalar advantageA​(y)A(y)dictates the magnitude of the update for responseyy. However, relying solely on this scalar signal can be insufficient, as samples with similar or even identical rewards may not be equally beneficial for robust policy improvement(Gupta et al.,2025).

To move beyond scalar rewards, we analyze the geometry of the policy’s representation space. Let𝐡​(y)∈ℝd\mathbf{h}(y)\in\mathbb{R}^{d}be the hidden representation of a rolloutyy. In our implementation,𝐡​(y)\mathbf{h}(y)is the last-layer hidden state of the final generated token, and is detached from the policy parameters. For any within-prompt pair of rollouts(yw,yl)(y_{w},y_{l})withr​(yw)>r​(yl)r(y_{w})>r(y_{l}), we define alatent displacement vector:

𝜹=𝐡​(yw)−𝐡​(yl).\boldsymbol{\delta}=\mathbf{h}(y_{w})-\mathbf{h}(y_{l}).(2)This vector𝜹\boldsymbol{\delta}serves as a directional proxy for policy improvement, pointing from a less-preferred to a more-preferred behavior. Our core thesis is that the geometric properties of these displacement vectors are key to understanding and mitigating RL instability.

3.2Directional Inconsistency

Policy updates are stable when per-rollout learning signals aredirectionally coherent. In online LLM RL, we can construct a forward-pass proxy for directional coherence by comparing rolloutswithin the same prompt: within a prompt, rollouts share the same instruction and differ primarily in the generated response, making reward-ordered comparisons less confounded by prompt-level variation.

For a preference pair(yw,yl)(y_{w},y_{l})withr​(yw)>r​(yl)r(y_{w})>r(y_{l}), the latent displacement direction𝐮=norm⁡(𝐡​(yw)−𝐡​(yl))\mathbf{u}=\operatorname{norm}(\mathbf{h}(y_{w})-\mathbf{h}(y_{l}))represents a reward-ordered behavioral change in representation space. Within an RL iteration these directions are not uniformly distributed: most concentrate around a dominant trend, while a small fraction exhibits large angular deviation. We refer to such severe angular outliers asdirectional inconsistency. These outliers can arise even among high-reward rollouts (e.g., reward-model artifacts and reward hacking(Shihab et al.,2025), or false positives from verifiable rewards where flawed reasoning accidentally reaches the correct answer(Lightman et al.,2024)), and can inject conflicting learning signals that amplify update variance. This motivates two research questions:

RQ 1(Observability).

How can we extract a reliableconsensusimprovement direction from noisy high-dimensional displacement representations?

RQ 2(Mitigation).

How can we neutralize severe directional outliers without expensive per-sample gradients or aggressively shrinking the effective batch?

Ourempirical analysisoffers affirmative insights. As shown in Sec.5.4(Fig.6), raw displacement vectors{𝜹}\{\boldsymbol{\delta}\}are directionally scattered, but a simple learned projector can reveal a highly concentrated directional structure, addressing RQ1. Furthermore, experiments in Sec.5.3demonstrate that injecting even a small fraction of reward noise, which creates directional outliers, leads to significant performance degradation, highlighting the practical importance of RQ2.

3.3Design Principles

The investigation of the above RQs motivates us to stabilize the online LLM RL with explicit directional calibration. We establish three design principles for a practical and effective solution:

1. 1.Forward-pass Only:The mechanism must avoid costly extra backward passes.
2. 2.Update Density Preservation:It should not discard prompts or significantly reduce batch size.
3. 3.Budgeted Intervention:The intervention must be conservative, targeting only the most severe outliers.

Theoretical Discussion.

Directional inconsistency can be interpreted through the variance of stochastic policy gradients. Let the policy objective beℒ​(θ)\mathcal{L}(\theta)and let the per-rollout gradient contribution be𝐠τ\mathbf{g}_{\tau}so that the population gradient is𝐠=𝔼​[𝐠τ]\mathbf{g}=\mathbb{E}[\mathbf{g}_{\tau}]. A standard source of instability is the variance of a mini-batch estimator𝐠^\hat{\mathbf{g}}:Var​(𝐠^)∝𝔼​[‖𝐠τ−𝐠‖22].\mathrm{Var}(\hat{\mathbf{g}})\ \propto\ \mathbb{E}\!\left[\|\mathbf{g}_{\tau}-\mathbf{g}\|_{2}^{2}\right].If two rollouts have comparable advantage magnitude but induce gradient contributions with opposing directions (i.e.,⟨𝐠τ1,𝐠τ2⟩<0\langle\mathbf{g}_{\tau_{1}},\mathbf{g}_{\tau_{2}}\rangle<0), then they can inflate the second moment of the estimator and increase the variability of updates.GeoAligndoes not compute𝐠τ\mathbf{g}_{\tau}explicitly; instead, it uses a forward-only proxy based on reward-ordered latent displacement directions. By identifying a batch-level consensus direction and conservatively neutralizing rollouts whose directions are strong angular outliers,GeoAlignaims to reduce the impact of such conflicting contributions and thereby improve training stability. A formal connection between latent directional consensus and gradient variance is an interesting direction for future work.

Figure 2:GeoAlignoverview.At each iteration, we form within-prompt preference displacements from low-reward to high-reward rollouts, project them onto a reward-sensitive manifold, construct a batch-wise consensus prototype, score rollouts by directional inconsistency, and rectify the experience buffer by replacing anomalous experiences with stable ones from the same prompt (equivalently, neutralizing their contribution) before running the RL update.

4GeoAlign: Online Calibration via Latent Directional Consistency

GeoAlignis a lightweight plug-in module forrollout curationinserted between rollout collection and the RL update. Given a batch of rollouts with rewards and forward-pass hidden states,GeoAlign(i) forms within-prompt preference pairs, (ii) learns an onlinedirectional projectorto reveal a concentrated “improvement manifold”, (iii) constructs ageometric prototypefor batch-wise consensus, (iv) scores rollouts by directional inconsistency, and (v) rectifies only the most severe outliers while preserving per-prompt rollout count.GeoAlignis illustrated in Fig.2and Algorithm1. The notations are summarized in Table5.

4.1Preference Pairs in Latent Space

We consider an online RL setup where, at each iteration, a policyπθ\pi_{\theta}generatesKKrollouts{yi,1,…,yi,K}\{y_{i,1},\dots,y_{i,K}\}for a promptxi∼𝒟x_{i}\sim\mathcal{D}, each receiving a scalar rewardri,kr_{i,k}. We extract the pooled hidden representation𝐡i,k∈ℝd\mathbf{h}_{i,k}\in\mathbb{R}^{d}for each rollout (e.g., from the last transformer layer).

Rather than analyzing individual points, we focus on thedirections of improvement. For each prompt, we construct a set of local preference pairs based on strict reward ranking:

𝒫=⋃i𝒫i,𝒫i={(k,ℓ)∣ri,k>ri,ℓ}.\mathcal{P}=\bigcup_{i}\mathcal{P}_{i},\qquad\mathcal{P}_{i}=\left\{(k,\ell)\mid r_{i,k}>r_{i,\ell}\right\}.(3)For any(i,k,ℓ)∈𝒫(i,k,\ell)\in\mathcal{P}, the raw difference vector𝜹i,k,ℓraw=(𝐡i,k−𝐡i,ℓ)\boldsymbol{\delta}^{\text{raw}}_{i,k,\ell}=(\mathbf{h}_{i,k}-\mathbf{h}_{i,\ell})

represents a transition from a lower-reward behavior to a higher-reward one. In high-dimensional language models, these raw directions can be entangled with task-irrelevant linguistic variance, motivating a projection step. For direction-consistency scoring, we use the unit displacement direction:𝐮i,k,ℓ=norm​(𝜹i,k,ℓraw)∈ℝd.\mathbf{u}_{i,k,\ell}=\mathrm{norm}\!\left(\boldsymbol{\delta}^{\text{raw}}_{i,k,\ell}\right)\in\mathbb{R}^{d}.

4.2Directional Distillation for Improvement Manifolds

The raw hidden representations of the policy are inherently entangled with task-irrelevant linguistic variances (e.g., lexical noise or stylistic bias), which can mask the underlying geometry of improvement.GeoAligntherefore learns a lightweight projectorℳψ:ℝd→ℝd′\mathcal{M}_{\psi}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d^{\prime}}that maps unit displacement directions to a lower-dimensional space where reward-ordered directions become more concentrated.

Within-Iteration Training Objective.Each preference pair yields a unit direction𝐮i,k,ℓ\mathbf{u}_{i,k,\ell}that ispositively orientedwith reward ordering (ri,k>ri,ℓr_{i,k}>r_{i,\ell}). Reversing the order corresponds to thenegativedirection−𝐮i,k,ℓ-\mathbf{u}_{i,k,\ell}. We train the projectorℳψ\mathcal{M}_{\psi}together with a temporary linear probew∈ℝd′w\in\mathbb{R}^{d^{\prime}}to classify the orientation:

ℒmap(ψ,w)=𝔼(i,k,ℓ)∈𝒫[log⁡(1+exp⁡(−w⊤​ℳψ​(𝐮i,k,ℓ)))+log(1+exp(+w⊤ℳψ(−𝐮i,k,ℓ)))].\small\begin{split}\mathcal{L}_{\mathrm{map}}(\psi,w)=\mathbb{E}_{(i,k,\ell)\in\mathcal{P}}\Big[&\log\!\bigl(1+\exp(-w^{\top}\mathcal{M}_{\psi}(\mathbf{u}_{i,k,\ell}))\bigr)\\ +\log\!\bigl(1&+\exp(+w^{\top}\mathcal{M}_{\psi}(-\mathbf{u}_{i,k,\ell}))\bigr)\Big].\end{split}(4)After the within-iteration update, we discard the probewwand keepℳψ\mathcal{M}_{\psi}for scoring.

Projected Improvement Directions.We compute normalized projected directions for each preference pair as𝐯i,k,ℓ=norm​(ℳψ​(𝐮i,k,ℓ)).\mathbf{v}_{i,k,\ell}=\mathrm{norm}\!\left(\mathcal{M}_{\psi}\big(\mathbf{u}_{i,k,\ell}\big)\right).Empirically, this projection is critical for revealing directional concentration and making a simple prototype robust (see Fig.6).

Implementation.ℳψ\mathcal{M}_{\psi}is a 3-layer MLP with ReLU. We update it for a small number of steps per RL iteration (50 steps; cosine LR schedule; initial LR10−310^{-3}) using only the current batch’s pairs. All representations are detached from the policy;GeoAligndoes not introduce additional backpropagation throughπθ\pi_{\theta}beyond the main RL update. More details are presented in AppendixC.1.

4.3Geometric Prototype and Anomaly Detection

With the projected manifold, we define the consensus improvement direction (geometric prototype) as the reward-margin-weighted normalized centroid of projected directions:

𝐯proto=norm​(∑(i,k,ℓ)∈𝒫(ri,k−ri,ℓ)⋅𝐯i,k,ℓ).\small\mathbf{v}_{\text{proto}}=\mathrm{norm}\left(\sum_{(i,k,\ell)\in\mathcal{P}}(r_{i,k}-r_{i,\ell})\cdot\mathbf{v}_{i,k,\ell}\right).(5)The reward-margin weighting ensures that preference pairs with larger reward differences contribute more strongly to the consensus direction, while unit-normalization ensures the prototype is determined by directional consensus rather than magnitudes.

Geometric Inconsistency Metric.For each preference pair, we define a cosine-distance deviation score as

si,k,ℓ=1−cos⁡(𝐯i,k,ℓ,𝐯proto)=1−⟨𝐯i,k,ℓ,𝐯proto⟩,s_{i,k,\ell}=1-\cos(\mathbf{v}_{i,k,\ell},\mathbf{v}_{\text{proto}})=1-\langle\mathbf{v}_{i,k,\ell},\mathbf{v}_{\text{proto}}\rangle,(6)where the second equality holds because both vectors are unit-normalized. We then define theGeometric Deviation Index (GDI)for a rolloutyi,ky_{i,k}as the cumulative deviation over all preference pairs that involve this rollout:

GDI​(i,k)=∑(a,b)∈ℐ​(i,k)si,a,b,\mathrm{GDI}(i,k)=\sum_{(a,b)\in\mathcal{I}(i,k)}s_{i,a,b},(7)whereℐ​(i,k)={(k,ℓ)∈𝒫i}∪{(ℓ,k)∈𝒫i}\mathcal{I}(i,k)=\{(k,\ell)\in\mathcal{P}_{i}\}\cup\{(\ell,k)\in\mathcal{P}_{i}\}. We use cumulative aggregation (rather than averaging) to emphasize rollouts that are consistently misaligned across many within-prompt comparisons; Sec.5.4validates that this form is more robust under reward noise.

As illustrated in Fig.3, GDI scores exhibit a characteristic long-tail distribution, implying that directional conflict concentrates in a sparse set of rollouts. We identify these high-GDI rollouts asdirectional outliers.

(a)Under a continuous reward model proxy. (b)With discrete binary rewards.

Figure 3:Anomaly score–rank distribution.Geometric inconsistency exhibits a long-tail pattern, motivating targeted intervention on a small subset of rollouts.

4.4Geometry-Aware Experience Rectification

GeoAlignmitigates the impact of directional outliers while preserving gradient density by rectifying experiences within each prompt. Motivated by the observed long-tail distribution (Fig.3), we employ an adaptive density-based strategy to identify the set of anomalous rollouts𝒴anom\mathcal{Y}_{\text{anom}}. Specifically, we model the probability density functionf^​(s)\hat{f}(s)of the GDI scores𝒮={GDI​(i,k)}i,k\mathcal{S}=\{\mathrm{GDI}(i,k)\}_{i,k}in the current batch via Kernel Density Estimation (KDE). A rollout is flagged as an outlier if its score lies in the high-deviation tail and its local density drops significantly below the peak:f^​(GDI​(i,k))<α⋅max⁡f^​(s)\hat{f}(\mathrm{GDI}(i,k))<\alpha\cdot\max\hat{f}(s), whereα∈(0,1)\alpha\in(0,1)controls the density threshold relative to the peak, with smaller values targeting only the most extreme deviations (sensitivity analysis in AppendixC.2).

This criterion enablesGeoAlignto automatically detect density collapses and isolate high-score anomalies that deviate from the collective improvement manifold. The remaining rollouts form the stable set𝒴stable\mathcal{Y}_{\text{stable}}.

Replacement within the same prompt.For each anomalous rollout in𝒴anom​(i)\mathcal{Y}_{\text{anom}}(i), we replace it in the experience buffer by sampling a representativey∗y^{*}from𝒴stable​(i)\mathcal{Y}_{\text{stable}}(i)following a Best-Random strategy(Zhang et al.,2025b). Let𝒴is:=𝒴stable​(i)\mathcal{Y}^{s}_{i}:=\mathcal{Y}_{\text{stable}}(i). Then:

P​(y∗=yj∣yj∈𝒴is)={0.5,if​yj=arg⁡maxy∈𝒴is⁡r​(y),0.5|𝒴is|−1,otherwise.\small P(y^{*}=y_{j}\mid y_{j}\in\mathcal{Y}^{s}_{i})=\begin{cases}0.5,&\text{if }y_{j}=\displaystyle\arg\max_{y\in\mathcal{Y}^{s}_{i}}r(y),\\[6.0pt] \dfrac{0.5}{|\mathcal{Y}^{s}_{i}|-1},&\text{otherwise}.\end{cases}(8) For group-normalized objectives (e.g., GRPO), prompt-wise replacement can be easily plug-and-play and viewed as adiscrete reweighting: it reduces the contribution of anomalous rollouts while increasing the mass on directionally consistent ones, without changing the number of rollouts per prompt. We discuss its effectiveness and ablate replacement versus zero-weighting and dropping in AppendixC.5.

4.5Practical Considerations

Ties and empty preference sets.If a prompt yields no strictly ordered preference pairs (e.g., binary rewards are tied), then𝒫i=∅\mathcal{P}_{i}=\emptysetandGeoAlignskips scoring that prompt, leaving its rollouts unchanged. If the batch yields|𝒫|=0|\mathcal{P}|=0,GeoAlignbecomes an identity mapping.

Complexity.Forming within-prompt preference pairs costsO​(N​K2)O(NK^{2})and scoring costsO​(|𝒫|​d′)O(|\mathcal{P}|d^{\prime})per iteration. WhenKKis large, one can subsample pairs per prompt (e.g., top-mmwinners vs. bottom-mmlosers) to reduce|𝒫||\mathcal{P}|without changing the rest ofGeoAlign. In our experiments (K≤16K\leq 16) we use all strictly ordered pairs. WithK≤16K\leq 16and a small projector, the overhead is negligible relative to rollout generation and the whole RL (profiling in AppendixC.6).

5Experiment

(a)Qwen3-1.7B on Math (b)Qwen3-4B on Math (c)Qwen3-1.7B on HH-RLHF (d)Qwen3-4B on HH-RLHF

Figure 4:Training dynamics on evaluation benchmarks.GeoAlignyields smoother learning curves and higher final evaluation scores than robust RL baselines on DAPO-Math (binary verified reward) and HH-RLHF (continuous RM reward).### 5.1Experimental Setup

We validate our method across diverse tasks, models’ capabilities, datasets, and reward structures. Full implementation details are provided in AppendixB.

Tasks and Datasets.We evaluateGeoAlignacross two distinct domains: mathematical reasoning (requiring multi-step logical deduction) and human preference alignment via RLHF. The training is performed on theDAPO-Math(Yu et al.,2025)andHH-RLHF(Bai et al.,2022)datasets.

Evaluation and Metrics.For Math, we assess performance on a wide range of benchmarks, including AIME24 & 25, AMC23, MATH500(Hendrycks et al.,2021), Minerva(Lewkowycz et al.,2022), and OlympiadBench(He et al.,2024), using theaverage accuracyover the final five evaluation checkpoints as the primary metric. For RLHF, we use Qwen-MAX as an LLM judge with position-swapped pairwise comparisons; we report theaverage score (0-1)andproxy reward model scoreon the HH-RLHF test set. Additionally, we present evaluation curves to visualize dynamics during training, with details in AppendixB.2.

Implementation.Our experiments are built upon the Trinity-RFT(Pan et al.,2025)framework, usingGRPO(Shao et al.,2024)for policy optimization on Qwen3-1.7B and Qwen3-4B models(Yang et al.,2025). For the math task, we apply RL directly to instruction-tuned models using a binary, verified reward with 16 rollouts per prompt. The RLHF task follows a Supervised Fine-tuning (SFT) then RL pipeline, leveraging a continuous reward from theArmoRM(Wang et al.,2024)reward model with 8 rollouts per prompt. ForGeoAlign, we useα=0.12\alpha=0.12on HH-RLHF andα=0.05\alpha=0.05on Math. Further details on the training process and hyperparameters can be found in AppendixB.3&C.2.

Baselines.We compare our approach with a range of strong baselines, including (1)Base-GRPO(Shao et al.,2024)vanilla GRPO algorithm by processing the training data sequentially; (2)PF-PPO(Zhang et al.,2025a), which filters rollouts using Best-vs-Random(BR)and Best-vs-Worst(BW)strategies to mitigate reward noise; (3)PAR(Fu et al.,2025), which reshapes rewards with a sigmoid function to prevent reward hacking; (4)PODS(Xu et al.,2025), which down-samples rollouts to maximize reward variance; and (5)Seed-GRPO(Chen et al.,2025), which uses semantic entropy to modulate policy updates for improved stability.

5.2Overall Performance

We first evaluateGeoAlign, the lightweight plug-in for rollout curation with minimal changes to the standard RL pipeline, assessing its effectiveness and stability.

Mathematical Reasoning.For the structured task of mathematical reasoning with binary verified rewards,GeoAlignshows robust and superior performance. Results are summarized in Table1and Fig.4(a, b). For Qwen3-1.7B,GeoAlignachieves the highest average score of40.44, outperformingBase-GRPOby1.6%relatively. This trend of consistent improvement continues on the larger Qwen3-4B model, whereGeoAligngains the top score of55.94, marking a relative gain of2.1%overBase-GRPOand outperforming all other robust methods. The evaluation curves in Fig.4further underscore this, showing thatGeoAlign’s performance trajectory is not only consistently higher but also more stable, leading to its higher final scores. In AppendixD.1, we present detailed results to showGeoAlign’s strong performance on challenging benchmarks like AIME25 and MATH500, indicating its ability to foster more coherent, multi-step reasoning. We also verify thatGeoAlignpreserves solution diversity via Pass@kkanalysis in AppendixC.4.

Table 1:Comparison of different baselines on Math, with complete per-benchmark performance detailed in Table13&14.Preference Alignment.As shown in Table2,GeoAlignconsistently outperforms all baselines on the HH-RLHF task across both Qwen3-1.7B and Qwen3-4B models. On the 1.7B model, our method achieves the highest mean score of0.8885, yielding a6.4%relative improvement over vanilla GRPO and surpassing the strongest robust baselinePAR. A similar trend holds for the 4B model, whereGeoAlignagain attains the top mean score of0.8894. The evaluation dynamics in Fig.4(c, d) further illustrate thatGeoAlign’s proxy reward (orange curve) follows a stable trajectory to a higher final value compared to the more clustered and ultimately lower performance of others.

Table 2:Comparison of different baselines on HH-RLHF, with complete helpfulness / harmfulness performance detailed in Table15&16.Training Dynamics.The consistent gains on evaluation benchmark across both discrete (Math) and continuous (HH-RLHF) reward settings underscore the versatility of our geometric approach (Fig.4). Beyond final scores,GeoAlignis designed to enhance training stability by mitigating high-variance updates from directionally inconsistent rollouts. To verify this, we performed an analysis of training dynamics across multiple seeds and optimization dynamics. As detailed in AppendixC.3,GeoAlignachieves positive performance gains while maintaining a stable and consistent optimization path.

5.3Robustness to Reward Corruption

A core motivation forGeoAlignis to build resilience against misspecified or noisy reward signals, which can be the primary source of directional inconsistency. To directly test this, we conduct controlled experiments byintentionally corruptingthe reward signals during training on Qwen3-1.7B. For the math task, we inject noise by randomly flipping the binary (0 / 1) rewards for a specified fraction of rollouts within each task. For HH-RLHF, we corrupt the rewards by reassigning them to their rollout-level extremes: responses scoring below the group mean are reassigned the maximum reward within the group, and vice versa.

Table 3:Evaluation performance with error injection, utilizing average score for Math and proxy reward for HH-RLHF as metrics.Table3reveals varying sensitivities to reward corruption across tasks. While a mere 5% error rate significantly degrades absolute accuracy in mathematical reasoning,GeoAlignmaintains the highest overall performance and a substantial margin overBase-GRPO, confirming its resilience to reward flipping.This robustness extends to the HH-RLHF task. While all methods see a performance decline under increasing noise,GeoAlignconsistently maintains thehighest proxy reward scoreat both 5% and 10% error levels, preserving its advantage over all baselines.

The visualization in Fig.5provides direct empirical validation for ourGDI scorein Eq.7as a reliable signal of reward noise. The plots show that the intentionally corrupted rollouts (black crosses) aredisproportionately concentrated in the high-GDI regionacross both tasks and noise levels. This confirms thatGeoAlign’s geometric metric effectively identifies and corrects rollouts whose learning signals conflict with the batch-wise consensus. This strong correlation between known reward errors and high GDI scores validatesGeoAlign’s ability to precisely target and neutralize sources of training instability.

Figure 5:Injected reward errors concentrate among high-GDI rollouts.Each panel ranks rollouts by GDI (blue line) for Math (left) and HH-RLHF (right). Black crosses denote rollouts whose rewards were corrupted, showing that injected errors are disproportionately located in the high-GDI region.(a) (b) (c) (d)

Figure 6:Visualization of latent vectors before (a, b) and after (c, d) projection. Raw vectors(a) are scattered, with widely dispersed cosine similarities to the centroid (b). After projection, latent vectors(c) become highly aligned, with nearly 90% achieving>0.8>0.8cosine similarity with the prototype (d). This demonstrates the projector’s role in enabling robust consensus detection.

5.4Analysis and Ablation Studies

The Central Role of the Directional Projector.

A key component ofGeoAlignis the online-trained projectorℳψ\mathcal{M}_{\psi}, designed to distill a coherent improvement manifold from noisy representations. To verify its role, we visualize the latent displacement vectors before and after projection in Fig.6. In the raw representation space (a-b), displacement directions are scattered andlack a clear consensus, with their cosine similarities to the batch centroid widely dispersed around zero. In stark contrast, after applying the projector (c-d), these vectors align along a dominant axis. As shown,nearly 90%of the projected vectors achieve a cosine similarity greater than 0.8 with the consensus prototype𝐯proto\mathbf{v}_{\text{proto}}, demonstrating that the projector is crucial for creating a geometrically coherent structure.

Furthermore, we confirm that the projector learns ageneralizable signalrather than overfitting to batch-specific artifacts. We analyze its validation accuracy on held-out preference pairs and its robustness to architectural changes (detailed in AppendixC.1). Our default lightweight projector already achieves high and stable validation accuracy across tasks with median>85%>85\%for HH-RLHF and>97%>97\%for Math, as shown in Fig.9. Furthermore, our ablations show that this high accuracy is maintained even when varying the projector’s depth in Table7, confirming thatℳψ\mathcal{M}_{\psi}captures an intrinsic geometric signal of improvement and justifies our use of a computationally efficient architecture.

Validating the Directional Inconsistency Hypothesis.

To examine the harm of directional inconsistency, we introduce anAnomaly-Amplifystrategy. This approach oversamples rollouts identified as extreme geometric outliers byGeoAlign, thereby artificially increasing the policy’s exposure to these directionally inconsistent gradients during the RL update.

Figure 7:Evaluation curve between different rollouts’ curation strategy onMath (left)andHH-RLHF (right).Table 4:Comparison of accumulation and normalized aggregation schemes, utilizing average score for Math and proxy reward for HH-RLHF.As shown in Table4, across both Math and HH-RLHF tasks, increasing the exposure to these geometric anomalies results in a discernible performance stagnation or regression compared to the vanillaBase-GRPO. For instance, on the Math task, the accuracy drops to 39.71, failing to surpass the baseline. In contrast,GeoAlignachieves a higher performance of 40.44 by neutralizing these conflicting signals. The evaluation dynamics in Fig.7further visualize this trend, suggesting that the baseline’s performance may be constrained by learning from directionally conflicting samples, and thatGeoAlignprovides a viable criterion to identify the subset of rollouts responsible for such training inefficiency.

Normalization: Analysis of Anomaly Score Aggregation.

To analyze the anomaly score (Eq.7) aggregation strategy, we compare our default cumulative scheme against a normalized (mean-based) alternative. As shown in Table4, the cumulative approach yields consistently higher performance on both Math and HH-RLHF tasks. We attribute this to the cumulative score’s ability to retain influence frequency, as it naturally amplifies the signal of rollouts that are consistently disruptive—a crucial distinction diluted by averaging.

(a)Normolized on Math (b)Cumulative on Math

Figure 8:Distribution ofnormalized (left)vs.cumulative (right)anomaly scores on the Math task. Cumulative scores provide a wider dynamic range, better distinguishing severe outliers.To further illustrate this, we explicitly inject 5% flip noise into the Math task. Fig.8supports that cumulative GDI enhances the contrast between outliers and normal samples, creating a distinct margin at the extreme rank, while the mean-based variant results in a more uniform distribution that obscures the boundary.

6Conclusion

This work identifiesdirectional inconsistencyas a key instability in online RL for LLM alignment: even with reasonable rewards, a small set of rollouts can induce update directions that conflict with the batch-level improvement signal, inflating variance and causing brittle training dynamics. We proposeGeoAlign, a lightweight, plug-and-play rollout curation method that estimates a batch consensus prototype in a projected space and intervenes only on strong directional outliers via within-prompt replacements. Empirically,GeoAlignimproves stability and final performance on both continuous and binary rewards, and remains robust under reward corruption. More broadly, our findings suggest thatdirectional consensuscan serve as an orthogonal reliability signal to reward magnitude, motivating future work on principled links to gradient variance, more robust/online consensus estimation, and integration with other alignment objectives and model classes.

Acknowledgments

This work was supported in part by the New Generation Artificial Intelligence-National Science and Technology Major Project (2025ZD0123003), and the National Natural Science Foundation of China Enterprise Innovation and Development Joint Fund (Artificial Intelligence Field) Key Support Projects (U25B2072).

Impact Statement

This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

References

• Bai et al. (2022)Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al.Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022.
• Casper et al. (2023)Casper, S., Davies, X., Shi, C., Krendl Gilbert, T., Scheurer, J., Rando Ramirez, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al.Open problems and fundamental limitations of reinforcement learning from human feedback.Transactions on Machine Learning Research, 2023.
• Chen et al. (2025)Chen, M., Chen, G., Wang, W., and Yang, Y.Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization.arXiv preprint arXiv:2505.12346, 2025.
• Cheng et al. (2025a)Cheng, J., Xiong, G., Qiao, R., Li, L., Guo, C., Wang, J., Lv, Y., and Wang, F.-Y.Stop summation: Min-form credit assignment is all process reward model needs for reasoning.arXiv preprint arXiv:2504.15275, 2025a.
• Cheng et al. (2025b)Cheng, Z., Hao, S., Liu, T., Zhou, F., Xie, Y., Yao, F., Bian, Y., Zhuang, Y., Dey, N., Zha, Y., et al.Revisiting reinforcement learning for llm reasoning from a cross-domain perspective.arXiv preprint arXiv:2506.14965, 2025b.
• Choe et al. (2024)Choe, S. K., Ahn, H., Bae, J., Zhao, K., Kang, M., Chung, Y., Pratapa, A., Neiswanger, W., Strubell, E., Mitamura, T., et al.What is your data worth to gpt? llm-scale data valuation with influence functions.arXiv preprint arXiv:2405.13954, 2024.
• Dai et al. (2025)Dai, Y., Huang, Y., Zhang, X., Wu, W., Li, C., Lu, W., Cao, S., Dong, L., and Li, S.Data efficacy for language model training, 2025.URLhttps://arxiv.org/abs/2506.21545.
• Dao et al. (2022)Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022.
• Fu et al. (2025)Fu, J., Zhao, X., Yao, C., Wang, H., Han, Q., and Xiao, Y.Reward shaping to mitigate reward hacking in rlhf.InICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025.
• Gao et al. (2023)Gao, L., Schulman, J., and Hilton, J.Scaling laws for reward model overoptimization.InInternational Conference on Machine Learning, pp. 10835–10866. PMLR, 2023.
• Gupta et al. (2025)Gupta, A., Tang, S., Song, Q., Zhu, S., Hong, J., Saha, A., Gupta, V., Lee, N., Kim, E., Zhu, S., et al.Alphapo: Reward shape matters for llm alignment.InForty-second International Conference on Machine Learning, 2025.
• He et al. (2024)He, C., Luo, R., Bai, Y., Hu, S., Thai, Z., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., et al.Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3828–3850, 2024.
• Hendrycks et al. (2021)Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J.Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021.
• Hu et al. (2025)Hu, Y., Wu, F., Ye, H., Forsyth, D., Zou, J., Jiang, N., Ma, J. W., and Zhao, H.A snapshot of influence: A local data attribution framework for online reinforcement learning.arXiv preprint arXiv:2505.19281, 2025.
• Jinnai et al. (2025)Jinnai, Y., Morimura, T., Ariu, K., and Abe, K.Regularized best-of-n sampling with minimum bayes risk objective for language model alignment.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 9321–9347, 2025.
• Kwon et al. (2023)Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I.Efficient memory management for large language model serving with pagedattention.InProceedings of the 29th symposium on operating systems principles, pp. 611–626, 2023.
• (17)Kydlíček, H.Math-Verify: Math Verification Library.URLhttps://github.com/huggingface/math-verify.
• Lewkowycz et al. (2022)Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al.Solving quantitative reasoning problems with language models.Advances in neural information processing systems, 35:3843–3857, 2022.
• Li et al. (2025a)Li, S., Li, S., Yang, Z., Zhang, X., Chen, G., Xia, X., Liu, H., and Peng, Z.Learnalign: Reasoning data selection for reinforcement learning in large language models based on improved gradient alignment.arXiv preprint arXiv:2506.11480, 2025a.
• Li et al. (2025b)Li, X., Zou, H., and Liu, P.Limr: Less is more for rl scaling.arXiv preprint arXiv:2502.11886, 2025b.
• Li et al. (2025c)Li, Y., Pan, Z., Lin, H., Sun, M., He, C., and Wu, L.Can one domain help others? a data-centric study on multi-domain reasoning via reinforcement learning.arXiv preprint arXiv:2507.17512, 2025c.
• Lightman et al. (2024)Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K.Let’s verify step by step.InInternational Conference on Learning Representations, volume 2024, pp. 39578–39601, 2024.
• Pan et al. (2025)Pan, X., Chen, Y., Chen, Y., Sun, Y., Chen, D., Zhang, W., Xie, Y., Huang, Y., Zhang, Y., Gao, D., et al.Trinity-rft: A general-purpose and unified framework for reinforcement fine-tuning of large language models.arXiv preprint arXiv:2505.17826, 2025.
• Papyan et al. (2020)Papyan, V., Han, X., and Donoho, D. L.Prevalence of neural collapse during the terminal phase of deep learning training.Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.
• Saunshi et al. (2019)Saunshi, N., Plevrakis, O., Arora, S., Khodak, M., and Khandeparkar, H.A theoretical analysis of contrastive unsupervised representation learning.InInternational conference on machine learning, pp. 5628–5637. PMLR, 2019.
• Shao et al. (2024)Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y. K., Wu, Y., and Guo, D.Deepseekmath: Pushing the limits of mathematical reasoning in open language models.CoRR, abs/2402.03300, 2024.
• Shihab et al. (2025)Shihab, I. F., Akter, S., and Sharma, A.Detecting and mitigating reward hacking in reinforcement learning systems: A comprehensive empirical study.arXiv preprint arXiv:2507.05619, 2025.
• Skalse et al. (2022)Skalse, J., Howe, N., Krasheninnikov, D., and Krueger, D.Defining and characterizing reward gaming.Advances in Neural Information Processing Systems, 35:9460–9471, 2022.
• Wang et al. (2024)Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T.Interpretable preferences via multi-objective reward modeling and mixture-of-experts.arXiv preprint arXiv:2406.12845, 2024.
• Wang & Isola (2020)Wang, T. and Isola, P.Understanding contrastive representation learning through alignment and uniformity on the hypersphere.InInternational conference on machine learning, pp. 9929–9939. PMLR, 2020.
• Xu et al. (2025)Xu, Y. E., Savani, Y., Fang, F., and Kolter, J. Z.Not all rollouts are useful: Down-sampling rollouts in llm reinforcement learning.arXiv preprint arXiv:2504.13818, 2025.
• Yang et al. (2024a)Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al.Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024a.
• Yang et al. (2025)Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025.
• Yang et al. (2024b)Yang, R., Ding, R., Lin, Y., Zhang, H., and Zhang, T.Regularizing hidden states enables learning generalizable reward model for llms.Advances in Neural Information Processing Systems, 37:62279–62309, 2024b.
• Yu et al. (2025)Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., et al.Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025.
• Zhang et al. (2025a)Zhang, C., Shen, W., Zhao, L., Zhang, X., Xu, X., Dou, W., and Bian, J.Policy filtration for rlhf to mitigate noise in reward models.InForty-second International Conference on Machine Learning, 2025a.
• Zhang et al. (2025b)Zhang, C., Shen, W., Zhao, L., Zhang, X., Xu, X., Dou, W., and Bian, J.Policy filtration for RLHF to mitigate noise in reward models.InForty-second International Conference on Machine Learning, 2025b.
• Zhao et al. (2023)Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al.Pytorch fsdp: experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023.

Appendix ANotation Summary

For ease of reading and reference, we present the symbols used in this paper in Table5.

Table 5:Table of notations used in this paper.

Appendix BDetails of Experimental Setup

B.1Experimental Tasks and Datasets

We validate our proposed method across two distinct domains to demonstrate its versatility: mathematical reasoning and reinforcement learning from human feedback (RLHF). The former assesses its capacity for rigorous, multi-step logical ability, while the latter tests the model’s ability to align with complex, subjective human preferences.

Mathematics

For mathematical reasoning, the objective is to enhance the model’s problem-solving capabilities. We use theDAPO-Math-17kdataset(Yu et al.,2025)for training. This dataset consists of 17,398 problems, primarily from competitive mathematics, with integer-based answers. The verified and binary reward structure (1.0 for a correct answer, 0.0 otherwise) provides a clear, objective signal for improving logical and computational accuracy.

RLHF

For the RLHF task, our goal is to fine-tune a model to be both helpful and harmless. The training is based on theHH-RLHFBai et al. (2022)dataset, which contains 144,716 training samples for SFT and 17,226 training examples for RL. Each example is a pair of responses (‘chosen‘ and ‘rejected‘) to a given prompt, reflecting human preferences. This dataset enables the model to learn nuanced distinctions between desirable and undesirable behaviors through RL.

B.2Evaluation and Metrics

Mathematics

We evaluate mathematical reasoning on a suite of benchmarks commonly used for assessing Qwen-Math series(Yang et al.,2024a)models. This selection covers a wide spectrum of difficulty, from challenging high-school problems to graduate-level and Olympiad-level questions:

• •AIME24/AIME25: The American Invitational Mathematics Examination (AIME) features highly challenging, competition-level problems for high-school students.
• •AMC23: A benchmark consisting of 40 problems from the American Mathematics Competitions (AMC). This widespread contest series for high-school students serves as a key test of foundational mathematical reasoning.
• •MATH500(Hendrycks et al.,2021): A classic and challenging benchmark comprising 500 problems from high-school math competitions, designed to test multi-step reasoning across algebra, combinatorics, geometry, number theory.
• •Minerva(Lewkowycz et al.,2022): A benchmark featuring 272 advanced problems from university and graduate-level STEM courses, covering subjects like differential equations and special relativity.
• •OlympiadBench(He et al.,2024): An Olympiad-level benchmark containing 675 exceptionally difficult problems in mathematics and science that demand profound and complex reasoning abilities.

The primary metric isaccuracy, calculated as the percentage of correctly solved problems. During training, we perform online evaluation at regular intervals (every 10 steps). Following common practice for mathematical reasoning tasks(Li et al.,2025c; Cheng et al.,2025b), we use a sampling temperature ofT=1.0T=1.0for generation, which introduces stochasticity. To mitigate this variance and obtain a reliable measure of the model’s performance near convergence, we report the final score as theaverage accuracy over the last five evaluation steps. In addition to final scores, we also present the accuracy evaluation curve on a validation set to show the dynamic performance during training, and using an Exponential Moving Average (EMA) with a factor of 0.6 for smoothing. To ensure a robust estimate on more difficult benchmarks, we follow standard protocols by generating multiple solutions per problem and averaging the outcomes: 8 generations for each problem in AIME 24/25 and 4 for AMC23.

RLHF

We evaluate RLHF performance using an LLM-as-a-Judge approach on the test set of HH-RLHF, which is split into ‘helpful‘ and ‘harmless‘ subsets. To compare our model against a baseline, we conduct pairwise evaluations arbitrated by a powerful external LLM (Qwen-MAX). To mitigate position bias, we perform a swap strategy: for each test prompt, we generate two comparisons, one with our model asModel Aand the baseline asB, and another with the roles reversed. In addition to final scores, we also present the proxy reward evaluation curve on a validation set to show the dynamic performance during training. The final metrics are derived from aggregating the results of these paired comparisons:

• •Win Rate:The percentage of times our model is judged superior to, equal to, or inferior to the baseline across all test prompts. A tie is declared if the judge’s decisions in the two swapped runs are contradictory.
• •Average Score:A normalized score between 0 and 1, averaged across all judgments, reflecting the absolute quality.
• •Proxy Reward:The score assigned by the reward model,ArmoRM(Wang et al.,2024), on a held-out validation set, used to track performance during training.

B.3Training Details

Framework and Infrastructure.

All experiments are conducted on theTrinity-RFT(Pan et al.,2025)framework, utilizing nodes equipped with 8 NVIDIA A100 (80GB) GPUs. We employ PyTorch’s Fully Sharded Data Parallel (FSDP)(Zhao et al.,2023)for efficient distributed training. During the exploration phase, response generation is accelerated using vLLM(Kwon et al.,2023)via flash-attention(Dao et al.,2022). We utilize GRPO(Shao et al.,2024)as the RL algorithm (detailed in AppendixB.4). We setenable_thinkingasFalsefor Qwen3 series to avoid long thinking paradigm.

Domain-Specific Training Processes.

Our experimental design validates our method across two distinct domains—RLHF and Mathematics—each with a unique training pipeline and reward structure.

• ForMathematics, we perform RL directly on theinstruction-tuned Qwen3 models(Yang et al.,2025)(1.7Band4B). The reward mechanism is binary: a response is awarded a reward of 1.0 if the answer extracted from the\boxed{}environment matches the ground truth, and 0.0 otherwise. This verification is performed using themath-verifylibrary(Kydlíček,), we employ a larger number of rollouts per prompt (n=16n=16). The training process is computationally demanding; a single run for the 1.7B model takes approximately20-22 hours, while the 4B model requires36-40 hours, both on a single8-GPU node.

• ForRLHF, we adopt a two-stage process. First, thebase Qwen3 models(Yang et al.,2025)(1.7B-Baseand4B-Base) undergo supervised fine-tuning (SFT) on the HH-RLHF dataset to align with the desired conversational style. Subsequently, these SFT models are further optimized using RL. The reward signal is continuous, obtained by querying an external reward modelArmoRM(Wang et al.,2024). Given the rich, continuous feedback from the reward model, a smaller number of rollouts per prompt (n=8n=8) is sufficient for effective policy updates. A single training process (SFT+RL) for the 1.7B model takes approximately12-13 hours, while the 4B model requires16-17 hours, both on a single8-GPU node.

Prompt.

We employ domain-specific prompt, with the exact templates shown bellow. For RLHF training, we use the conversation from the dataset directly as the input prompt. For evaluation, a distinct LLM-as-a-Judge prompt is used. In mathematics, a consistent prompt format is applied to both training and evaluation.

Mathematics Training & Evaluation PromptSystem_Prompt:Please reason step by step, and put your final answer within \\boxed{}.User_Prompt:{task_text}

RLHF Evaluation PromptYou are an impartial judge.Compare two responses for helpfulness and harmlessness.Choose the better one.You MUST select a winner unless the two responses are nearly identical.If one is even slightly better, choose it.“Tie” should be extremely rare.Return ONLY a JSON object:{“overall_winner”: “model_a” | “model_b” | “tie”,“model_a_score”: <float 0-1>,“model_b_score”: <float 0-1>}The winner must have a strictly higher score.

Hyperparameters.

The core training configurations are tailored for two model sizes, while many parameters are shared, key differences exist in learning rates, batch sizes, and rollout numbers to accommodate the distinct requirements of each domain and model scale. A comprehensive summary of hyperparameters for all six settings is provided in Table6.

Table 6:Key hyperparameters for SFT and RL across different models and domains. ”Inst.” is an abbreviation for Instruct-tuned models.

B.4Policy Optimization Algorithm: GRPO

For policy optimization, we employGroup Relative Policy Optimization (GRPO)(Shao et al.,2024), a policy gradient algorithm notable for operating without a learned value function. Instead, it derives the advantage signal by contextualizing a response’s reward within a group of peers. The process begins by sampling a group ofGGresponses,Y={y1,…,yG}Y=\{y_{1},\dots,y_{G}\}, from the current policyπθold\pi_{\theta_{\text{old}}}for a given promptxx. The meanμY\mu_{Y}and standard deviationσY\sigma_{Y}of the rewards for this group,{R​(yk)}k=1G\{R(y_{k})\}_{k=1}^{G}, are then computed. The advantageA​(yk)A(y_{k})for each responseyky_{k}is its normalized reward:

A​(yk)=R​(yk)−μYσY.A(y_{k})=\frac{R(y_{k})-\mu_{Y}}{\sigma_{Y}}.(9)The policy parametersθ\thetaare updated by optimizing a clipped surrogate objective. This objective relies on the probability ratiorθ​(y)=πθ​(y|x)πθold​(y|x)r_{\theta}(y)=\frac{\pi_{\theta}(y|x)}{\pi_{\theta_{\text{old}}}(y|x)}, which compares the likelihood of a response under the new and old policies. The GRPO objective function,ℒGRPO​(θ)\mathcal{L}_{\text{GRPO}}(\theta), incorporates this ratio and the calculated advantage in a form analogous to PPO:

ℒGRPO​(θ)=𝔼y∼πθold​[min⁡(rθ​(y)​A​(y),clip​(rθ​(y),1−ϵ,1+ϵ)​A​(y))].\mathcal{L}_{\text{GRPO}}(\theta)=\mathbb{E}_{y\sim\pi_{\theta_{\text{old}}}}\left[\min\left(r_{\theta}(y)A(y),\text{clip}(r_{\theta}(y),1-\epsilon,1+\epsilon)A(y)\right)\right].(10)Here, theclip​(rθ​(y),1−ϵ,1+ϵ)\text{clip}(r_{\theta}(y),1-\epsilon,1+\epsilon)term constrains the probability ratio, ensuring that policy updates are kept within a stable, trusted region to prevent destructively large steps during optimization.

B.5Baselines

We compare our method against several recent and representative baselines that aim to improve RL by addressing common challenges such as reward noise, reward hacking, and inefficient sampling. The configurations for each baseline are detailed:

• •Base-GRPO(Shao et al.,2024): The vanilla implementation of the GRPO algorithm. In this baseline, tasks are sampled sequentially from the training dataset without any curriculum or specialized selection strategy. It serves as our primary baseline to demonstrate the performance of the standard policy optimization.
• •PF-PPO(Zhang et al.,2025a): Policy Filtration for PPO (PF-PPO) mitigates the impact of noisy reward signals by filtering out rollouts with intermediate reward scores, which are often the most unreliable. It selectively samples rollouts for PPO updates from the extremes of the reward distribution. We experiment with two of its variants:Best-vs-Random (BR), which samples from the highest-rewarded and a random selection of the rest, andBest-vs-Worst (BW), which samples from the highest and lowest rewarded rollouts.
• •PAR(Fu et al.,2025): Policy-Agnostic Reward shaping (PAR) is designed to prevent reward hacking by reshaping the reward signal. It centralizes rewards relative to a reference model’s performance and applies asigmoid transformation. This process clips and rescales extreme reward values, discouraging the policy from exploiting inaccuracies in the reward model to achieve artificially high scores.
• •PODS(Xu et al.,2025): Policy-Oriented Down-Sampling (PODS) addresses inefficient sampling by selecting a more informative subset of rollouts for training. Its core strategy,Variance Maximization, first prioritizes high-reward samples and then selects additional samples from the remaining pool to maximize the reward variance of the final training batch.
• •Seed-GRPO(Chen et al.,2025): Semantic Entropy Enhanced GRPO (Seed-GRPO) improves training stability by dynamically adjusting policy updates based on uncertainty. It calculates thesemantic entropy (SE)of rollouts as a proxy for uncertainty. For high-entropy (high-uncertainty) problems, it applies a more conservative update, while for low-entropy problems, it uses the standard learning signal, thus balancing exploration and exploitation.

Appendix CComprehensive Experiments

In this section, we provide a series of additional experiments to further analyzeGeoAlign’s behavior, justify our design choices, and verify its broader applicability and efficiency.

C.1Ablation on Projector Architecture

To analyze the stability and architectural sensitivity of the directional projectorℳψ\mathcal{M}_{\psi}, we evaluate its learning dynamics and ablate its depth. As shown in Fig.9, the projector learns a highly generalizable signal. The validation accuracy curves (a) demonstrate rapid convergence without signs of overfitting across all tasks and models, while the box plots (b) confirm consistently high median validation accuracies: typically>85%>85\%for HH-RLHF and>97%>97\%for Math. Leveraging this rapid convergence,GeoAlignemploys an early-stopping mechanism: projector training is halted once validation accuracy surpasses a threshold (95% for Math and 85% for HH-RLHF) to further reduce computational overhead.

We further ablate the number of MLP layers in Table7. The results reveal that validation accuracy is remarkably stable across different depths (from 2 to 5 layers), while the time overhead increases proportionally with architectural complexity. This confirms that a simple MLP is sufficient to capture the underlying geometric structure. We select the 3-layer architecture as our default as it provides an optimal trade-off between robust performance and minimal computational cost, reinforcingGeoAlign’s efficiency.

(a)Projector training dynamics (b)Accuracy Distribution on Validation Set

Figure 9:The projector learns a robust signal quickly and effectively. (a) Training and validation accuracy curves show rapid convergence without overfitting across all settings. (b) Box plots confirm consistently high median validation accuracy throughout training.Table 7:Ablation on projector depth. The results show that validation accuracy is highly stable across different architectures, while computational overhead (Time) slightly increases with depth.Table 8:Performance under differentGeoAligndensity sensitivities (α\alpha) on HH-RLHF. Table 9:Performance under differentGeoAligndensity sensitivities (α\alpha) on Math.

C.2Hyperparameter Sensitivity Analysis

To analyze the sensitivity ofGeoAlignto its main hyperparameter, we focus on the density thresholdα\alpha, which determines the sensitivity to density collapses for anomaly identification. Tables9and9present the performance across differentα\alphavalues for both HH-RLHF and Math tasks.

For HH-RLHF, characterized by inherent reward noise, performance peaks in the region ofα=0.12\alpha=0.12. For the deterministic Math task, a smaller threshold aroundα=0.05\alpha=0.05is found to be more effective. However, an excessively largeα\alphaleads to a discernible narrowing of these gains. Asα\alphaincreases beyond the task-specific peak, the performance advantage over the baseline diminishes. We attribute this to over-regularization: whileGeoAlignconcentrates updates along the dominant improvement direction, excessive density sensitivity enforces overly strict adherence to the batch-wise consensus. This risks marginalizing valid trajectories that exhibit high variance—such as distinct but correct reasoning paths that naturally deviate from the high-density region. Thus, a calibratedα\alphais essential to strike the trade-off between directional consistency (noise reduction) and manifold diversity (exploration).

The results indicate that the optimal thresholdα\alphacorrelates with the intrinsic noise level of the task; lower-noise environments favor more conservative density filtering. This analysis suggests that adaptively settingα\alphabased on a running estimate of reward variance constitutes a viable future direction.

C.3Analysis of Training Stability

Stability Across Random Seeds.

To assess the training stability ofGeoAlignand validate that its performance improvements are consistent, we conducted a targeted analysis across multiple random seeds. Due to thesignificant computational cost of full end-to-end training runs for LLMs(detailed in AppendixB.3), we adopted a focused approach to investigate stability. Specifically, we performed three independent runs with different random seeds for bothGeoAlignand theBase-GRPObaseline on the Qwen3-1.7B model. We focused on the initial 100 training steps, as this phase is critical for establishing a stable optimization trajectory.

Fig.10illustrates the training dynamics for both the mathematical reasoning (DAPO-Math) and preference alignment (HH-RLHF) tasks. For mathematical reasoning, we plot therollout/accuracy/meanandcritic/advantages/mean. For preference alignment, we display theeval/proxy_reward/meanand thecritic/advantages/mean. In all plots, the curves for the three independent runs are tightly clustered for bothGeoAlign(red) and the baseline (black). This demonstrates thatGeoAligndoes not introduce additional variance into the training process; instead, it maintains a highly stable and predictable learning path, which is consistent with its design objective of reducing destabilizing updates.

(a)Math - rollout/accuracy (b)Math - critic/advantages (c)HH - rollout/proxy_reward (d)HH - critic/advantages

Figure 10:Training dynamics over the first 100 steps forGeoAlign(red) andBase-GRPO(black) across three random seeds on Qwen3-1.7B. The tight clustering of curves for both methods highlights the stability of the training process, withGeoAlignconsistently following a stable trajectory.Furthermore, to quantify the stability of evaluation outcomes, we report the mean and standard deviation of the final evaluation scores across these three partial runs (at step 100). Table10summarizes these results. Although the number of runs is limited, the low standard deviation observed forGeoAlignprovides additional evidence of its robustness to initialization and data sampling randomness.

Table 10:Mean and standard deviation of evaluation scores across 3 random seeds for Qwen3-1.7B. For Math, the score is the average accuracy at step 100. For HH-RLHF, it is the average proxy reward model score at step 100.

Optimization Dynamics.

To further substantiate the stability gains, we monitor two PPO trust-region diagnostics throughout training: the policy KL divergence, which measures the distributional shift between the updated and reference policy, and the clipping fraction, which captures the fraction of updates clipped by the trust-region constraint. Both serve as indicators of extreme parameter updates that may destabilize the optimization trajectory. Fig.11shows the corresponding training curves, and Table11summarizes the mean and median values aggregated over the full training run on both tasks.GeoAlignconsistently reduces both quantities across Math and HH-RLHF, indicating that directional rectification leads to fewer trust-region violations and a more controlled optimization trajectory.

Table 11:PPO trust-region diagnostics (mean and median over full training).GeoAlignreduces both KL divergence and clipping fraction on both tasks.(a)PPO KL (Math) (b)Clip Fraction (Math) (c)PPO KL (HH-RLHF) (d)Clip Fraction (HH-RLHF)

Figure 11:PPO KL divergence and clipping fraction throughout training on Qwen3-1.7B across three random seeds.

C.4Solution Diversity: Pass@k Analysis

A natural concern is whether penalizing directional outliers suppresses valid alternative reasoning paths. We clarify thatGeoAligndoes not judge solution creativity directly. A correct but unconventional solution gains high reward and acts as a positive anchor in a preference pair, defining the target endpoint of an improvement direction. The consensus prototype (Eq. 5) aggregates these directions across pairs, so correct but creative solutions naturally shape the consensus. Rollouts are flagged as outliers only when their implied directions contradict this collective flow, typically due to reward hacking or logical inconsistency rather than genuine novelty.

To directly test whetherGeoAlignsuppresses solution diversity, we reportPass​@​k=1−(n−ck)/(nk)\text{Pass}@k=1-\binom{n-c}{k}/\binom{n}{k}on the three most challenging benchmarks in our evaluation suite (AIME24, AIME25,n=64n=64; AMC23,n=32n=32), where low baseline accuracy makes Pass@kkmost sensitive to changes in solution diversity. As shown in Fig.12,GeoAlignmaintains a Pass@kkslope equal to or steeper thanBase-GRPOacross allkkvalues, confirming that the model’s capacity to discover diverse correct solutions is fully intact and thatGeoAlignintervenes on directional outliers rather than valid alternative reasoning modes.

(a)AIME 2024 (b)AIME 2025 (c)AMC 2023

Figure 12:Pass@kkcurves on AIME24/25 (n=64n=64) and AMC23 (n=32n=32).GeoAlignmaintains a slope equal to or steeper thanBase-GRPOacross allkk, confirming that solution diversity is preserved.

C.5Ablation on Rectification Strategy

We ablate the rectification mechanism in Sec. 4.4 by comparing three strategies for handling anomalous rollouts:Replacement(the default inGeoAlign, which substitutes anomalous rollouts with stable within-prompt alternatives),Zero-weighting(setting the advantage of anomalous rollouts to zero), andDropping(discarding anomalous rollouts entirely). Results are shown in Table12.

The results reveal a hierarchy: Replacement>>Zero-weighting>>Dropping. Dropping performs worst as it reduces the per-prompt rollout countKK, directly altering the advantage normalization denominator in GRPO and injecting artificial variance. Zero-weighting preservesKKbut passively neutralizes anomalous rollouts, reducing effective gradient density. Replacement preserves bothKKand full gradient density by substituting anomalies with directionally consistent alternatives, yielding the best performance on both tasks.

Table 12:Ablation on rectification strategy on Qwen3-1.7B. Math reports average accuracy over the last five evaluation checkpoints; HH-RLHF reports the gain in proxy reward score over the SFT baseline.

C.6Computational Overhead

We empirically quantify the computational overhead ofGeoAlignby measuring its impact on GPU memory usage and training throughput. Our analysis indicates that the overheadis minimal. In terms of memory, theGeoAlignmodule, including its online projector and associated computations, consumesan additional 300–500 MB of GPU memory. This constitutes a negligible increase relative to the memory required by the base model. For temporal cost, we measured the end-to-end wall-clock time. As detailed in our ablation studies (Table7), the operations specific toGeoAlignaddapproximately 1%to the total training time per step.

This low overhead is expected, asGeoAlign’s core operations—a few matrix multiplications for the projector and vector arithmetic for the consensus prototype—are highly efficient on modern GPUs and operate on low-dimensional hidden states detached from the computation graph. This confirms thatGeoAlignis a lightweight plug-in that can be integrated into standard RL pipelines with negligible performance impact.

Appendix DComplete Experimental Results

D.1Complete Results of Overall Performance on Math

We present the complete experimental results of Qwen3-1.7B and Qwen3-4B on mathematic task in the validation experiments in Table13and Table14respectively, demonstrating the robustness and effectiveness of our approach, consistently achieving the highest average scores across different benchmarks.

Table 13:Comparison of different methods on 1.7B-MathTable 14:Comparison of different methods on 4B-Math

D.2Complete Results of Overall Performance on Math

We present the complete scores of Qwen3-1.7B and Qwen3-4B on HH-RLHF task in the validation experiments in Tables15. Additionally, we display the origin win-rate comparison of our method and baselines in Table16. Both tables demonstrate the robustness and effectiveness of our approach, consistently achieving the highest helpful and harmless scores across models.

Table 15:Evaluation scores of different models on the HH-rlhf datasetTable 16:Evaluation win-rate of different models on the HH-rlhf dataset

Appendix EAlogorithm and Pseudocode ofGeoAlign

Algorithm1summarizes the full procedure.

Algorithm 1GeoAlign: Online Policy Training with Latent Directional Rectification1:Initialize:policy

πθ\pi_{\theta}, reward function

r​(⋅)r(\cdot), projector

ℳψ\mathcal{M}_{\psi}.

2:foriteration

t=1,2,…t=1,2,\dotsdo

3:Data Collection:For each prompt

xix_{i}in a batch, generate

KKrollouts

{yi,k}\{y_{i,k}\}, compute rewards

ri,kr_{i,k}, and extract detached hidden states

{𝐡i,k}\{\mathbf{h}_{i,k}\}.

4:Assemble experience batch

ℬ={(xi,yi,k,ri,k,𝐡i,k)}i,k\mathcal{B}=\{(x_{i},y_{i,k},r_{i,k},\mathbf{h}_{i,k})\}_{i,k}.

5:Experience Rectification:

6:

ℬrect←GeoAlign​(ℬ,ℳψ)\mathcal{B}_{\text{rect}}\leftarrow\textsc{GeoAlign}(\mathcal{B},\mathcal{M}_{\psi})⊳\trianglerightLightweight, plug-and-play curation module.

7:Policy Update:

8:

θ←RL_Update​(θ,ℬrect)\theta\leftarrow\textsc{RL\_Update}(\theta,\mathcal{B}_{\text{rect}})⊳\trianglerightUse rectified batch for any policy gradient update.

9:endfor

10:functionGeoAlign(

ℬ,ℳψ\mathcal{B},\mathcal{M}_{\psi})

11:1. Construct Preference Pairs:For each prompt

xix_{i}, form pair indices

𝒫i={(k,ℓ)∣ri,k>ri,ℓ}\mathcal{P}_{i}=\{(k,\ell)\mid r_{i,k}>r_{i,\ell}\}. Aggregate into a batch-wide set

𝒫=⋃i𝒫i\mathcal{P}=\bigcup_{i}\mathcal{P}_{i}(Eq.3).

12:if

|𝒫|=0|\mathcal{P}|=0then

13:return

ℬ\mathcal{B}⊳\trianglerightSkip if no strict preferences exist (e.g., all rewards tied).

14:endif

15:2. Update Directional Projector:For a fixed number of steps, update projector

ℳψ\mathcal{M}_{\psi}and a temporary probe

wwby minimizing

ℒmap​(ψ,w)\mathcal{L}_{\mathrm{map}}(\psi,w)(Eq.4) on the unit directions derived from

𝒫\mathcal{P}. Discard

wwafter training.

16:3. Compute Projected Directions:For each pair

(i,k,ℓ)∈𝒫(i,k,\ell)\in\mathcal{P}:

17:

𝐮i,k,ℓ←norm​(𝐡i,k−𝐡i,ℓ)\mathbf{u}_{i,k,\ell}\leftarrow\mathrm{norm}(\mathbf{h}_{i,k}-\mathbf{h}_{i,\ell})⊳\trianglerightRaw unit direction.

18:

𝐯i,k,ℓ←norm​(ℳψ​(𝐮i,k,ℓ))\mathbf{v}_{i,k,\ell}\leftarrow\mathrm{norm}(\mathcal{M}_{\psi}(\mathbf{u}_{i,k,\ell}))⊳\trianglerightProjected unit direction.

19:4. Establish Consensus Prototype:Compute the batch consensus direction

𝐯proto\mathbf{v}_{\text{proto}}(Eq.5).

20:5. Score Rollout Inconsistency:

21:For each pair, compute deviation

si,k,ℓ=1−⟨𝐯i,k,ℓ,𝐯proto⟩s_{i,k,\ell}=1-\langle\mathbf{v}_{i,k,\ell},\mathbf{v}_{\text{proto}}\rangle(Eq.6).

22:For each rollout, aggregate pair deviations into a Geometric Deviation Index,

GDI​(i,k)\mathrm{GDI}(i,k)(Eq.7).

23:6. Identify Anomalies:Let

𝒮={GDI​(i,k)}i,k\mathcal{S}=\{\mathrm{GDI}(i,k)\}_{i,k}be all GDI scores. Estimate the PDF

f^​(s)\hat{f}(s)of GDI scores

𝒮\mathcal{S}via KDE. Flag

yi,ky_{i,k}as anomalous if its local density drops below the adaptive threshold:

24:

f^​(GDI​(i,k))<α⋅maxs∈𝒮⁡f^​(s)\hat{f}(\mathrm{GDI}(i,k))<\alpha\cdot\max_{s\in\mathcal{S}}\hat{f}(s).

25:Let

𝒴anom\mathcal{Y}_{\text{anom}}and

𝒴stable\mathcal{Y}_{\text{stable}}be the sets of anomalous and stable rollouts.

26:7. Rectify Experience Batch:Initialize

ℬrect←ℬ\mathcal{B}_{\text{rect}}\leftarrow\mathcal{B}.

27:foreach anomalous rollout

yi,k∈𝒴anomy_{i,k}\in\mathcal{Y}_{\text{anom}}do

28:Sample a replacement

y∗y^{*}from the stable set of the same prompt,

𝒴stable​(i)\mathcal{Y}_{\text{stable}}(i).

29:Replace the entry for

yi,ky_{i,k}with the entry for

y∗y^{*}in

ℬrect\mathcal{B}_{\text{rect}}.

30:endfor

31:return

ℬrect\mathcal{B}_{\text{rect}} 32:endfunction