N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization
Summary
N-GRPO introduces semantic neighbor mixing in the GRPO framework to enhance mathematical reasoning diversity while preserving semantic consistency, achieving improvements on math benchmarks and out-of-distribution tasks.
View Cached Full Text
Cached at: 06/12/26, 02:52 AM
Paper page - N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization
Source: https://huggingface.co/papers/2606.10768
Abstract
N-GRPO, a novel exploration strategy within GRPO framework, enhances mathematical reasoning in large language models through semantic neighbor mixing that maintains semantic consistency while injecting diversity.
The success ofLarge Language Modelsinmathematical reasoningrelies heavily on the generation of diverse and valid solution paths during therollout phase. However, current rollout techniques face a fundamental trade-off:token-level samplingoften yields redundant trajectories that differ only in rephrasing, whileembedding-level methodsutilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into theGroup Relative Policy Optimization(GRPO) framework. Rather than relying ontoken-level samplingor native embedding-level noise, our approach leveragesSemantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the localsemantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines onmath reasoning benchmarksbut also exhibits robust generalization capabilities onout-of-distribution tasks.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.10768
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.10768 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.10768 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.10768 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Multi-module GRPO: Composing Policy Gradients and Prompt Optimization for Language Model Programs
The paper introduces mmGRPO, a multi-module extension of Group Relative Policy Optimization (GRPO) that improves accuracy in modular AI systems by optimizing language model calls and prompts. It reports an average 11% accuracy improvement across various tasks and provides an open-source implementation in DSPy.
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
BiasGRPO proposes a framework using Group Relative Policy Optimization (GRPO) to stabilize social bias mitigation in LLMs by normalizing rewards across sampled completions, outperforming DPO and PPO on multiple benchmarks. The authors also release a compute-efficient bias reward model designed for integration into multi-objective RLHF pipelines.
Group Entropy-Controlled Policy Optimization
This paper proposes Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy to perform entropy-conditioned asymmetric advantage shaping, addressing heterogeneous entropy regimes across tasks during RL-based alignment of LLMs. Experiments show consistent improvements over GRPO and recent entropy-controlled methods across multiple benchmarks.
Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting
This paper proposes a covariance-aware variant of Group Relative Policy Optimization (GRPO) that uses Gaussian-kernel advantage reweighting to stabilize training entropy and improve reasoning performance in large language models.
F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking
F-GRPO proposes a factorized group-relative policy optimization framework that unifies candidate generation and ranking in a single autoregressive LLM, addressing credit assignment issues and improving top-ranked performance across sequential recommendation and multi-hop QA benchmarks.