PPO-HSC: An Exploratory Reinforcement Learning Framework Based on Wide-Area Policy Coverage Optimization

# PPO-HSC: An Exploratory Reinforcement Learning Framework Based on Wide-Area Policy Coverage Optimization
Source: [https://arxiv.org/html/2607.16206](https://arxiv.org/html/2607.16206)
11institutetext:School of Mathematics,
Hunan University, Changsha 410082, China22institutetext:College of Computer Science and Electronic Engineering,
Hunan University, Changsha 410082, China
22email:jayshum@hnu\.edu\.cn, hwchen@hnu\.edu\.cn###### Abstract

This paper introduces PPO\-HSC \(Proximal Policy Optimization with High\-order Sampling Coverage\), an exploratory reinforcement learning framework designed to address the "Invisible Shackles" of mode collapse in Large Language Model \(LLM\) fine\-tuning\. While standard Reinforcement Learning from Verifiable Rewards \(RLVR\) effectively reinforces high\-reward trajectories, it often leads models to over\-optimize known solutions, sacrificing curiosity and the ability to explore broader solution manifolds\. To overcome this, PPO\-HSC incorporates a High\-order Sampling Coverage \(HSC\) reward that incentivizes the discovery of "low\-similarity yet high\-validity" reasoning patterns\. By maintaining a dynamic trajectory library of verified unique solutions, the framework provides a differentiable signal that rewards semantic novelty while ensuring structural rationality through a plausibility constraint\. Empirical evaluations on mathematical reasoning \(GSM8K, SVAMP\) and code generation tasks demonstrate that PPO\-HSC significantly enhances solution diversity and state\-space coverage while maintaining or surpassing the accuracy and syntax integrity of state\-of\-the\-art RL baselines\. Our code is available at[https://github\.com/JJayshum/PPO\-HSC](https://github.com/JJayshum/PPO-HSC)\.

## 1Introduction

Large Language Models \(LLMs\) have demonstrated transformative potential in complex reasoning tasks, ranging from mathematical problem\-solving to automated code generation\. To further align these models with human intent and optimize performance, Reinforcement Learning \(RL\) techniques—most notably Proximal Policy Optimization \(PPO\)\[[1](https://arxiv.org/html/2607.16206#bib.bib1)\]and Reinforcement Learning from Verifiable Rewards \(RLVR\)\[[3](https://arxiv.org/html/2607.16206#bib.bib3)\]—have become the factual standard\. By iteratively reinforcing high\-reward trajectories, these frameworks enable LLMs to converge toward optimal solutions in vast decision spaces\.

Despite their success, current RL\-based fine\-tuning paradigms suffer from a critical limitation we term "Invisible Shackles\." Standard RL algorithms are inherently greedy; they tend to over\-optimize known high\-reward paths, leading to severe mode collapse\. In this regime, the model acts merely as an "accelerator of known solutions" rather than a creative reasoner\. It gravitates toward a narrow subset of the strategy space that yields immediate rewards, thereby losing the curiosity to explore the broader, potentially superior, or more diverse solution manifolds\. This lack of exploratory pressure prevents the model from discovering novel reasoning chains or diverse logic paths, which are essential for robust generalization\.

To break these shackles, we introduce PPO\-HSC \(Proximal Policy Optimization with High\-order Sampling Coverage\)\. The core philosophy of PPO\-HSC is to augment the objective function by shifting the focus from simple path reinforcement to boundary expansion\. Specifically, we incorporate a High\-order Sampling Coverage \(HSC\) reward alongside the standard objective rewards\. By maintaining a dynamic trajectory library derived from the base model, the HSC mechanism assigns intrinsic rewards to "low\-similarity yet high\-validity" trajectories\. This ensures that the model is incentivized to explore novel reasoning patterns that are both distinct from previously seen patterns and logically sound\.

![Refer to caption](https://arxiv.org/html/2607.16206v1/Comparison_of_Policy_Coverage_between_Standard_PPO_and_PPO_HSC.png)Figure 1:Comparison of Policy Coverage between Standard PPO and PPO\-HSCStandard PPO \(Mode\-Seeking\)\. Standard algorithms exhibit "mode\-seeking" behavior , gravitating toward singular path exploitation\. This leads to mode collapse, where the model merely accelerates known solutions rather than exploring the broader strategy space\.PPO\-HSC \(Wide\-Area Coverage\): Our framework shifts the focus to boundary expansion by rewarding "low\-similarity yet high\-validity" trajectories\. This enables wide\-area coverage across the reasoning manifold\.Mechanism: By maintaining a Dynamic Trajectory Library𝒟\\mathcal\{D\}, PPO\-HSC quantifies the novelty of generated sequences relative to historically known experiences\. Outcome: The model is incentivized to discover diverse, logically sound reasoning chains while maintaining high accuracy\.#### Contributions

The contributions of this work are three\-fold:\(i\)A Novel Framework: We propose PPO\-HSC, an exploratory reinforcement learning framework that redefines the optimization target from singular path exploitation to comprehensive sampling space coverage\.\(ii\)HSC Reward Mechanism: We design a high\-order sampling coverage reward based on a dynamic trajectory library\. This mechanism effectively quantifies the novelty and structural rationality of generated sequences, providing a fine\-grained signal for exploration\.\(iii\)Empirical Validation: Extensive experiments on diverse multi\-solution reasoning tasks demonstrate that PPO\-HSC significantly enhances solution diversity and state\-space coverage while maintaining or even surpassing the accuracy of state\-of\-the\-art RL baselines\.

## 2Related Works

Reinforcement Learning \(RL\) has emerged as a cornerstone for aligning Large Language Models with human values and complex reasoning requirements\. Representative frameworks such as Proximal Policy Optimization \(PPO\) \[1\] and Direct Preference Optimization \(DPO\)\[[2](https://arxiv.org/html/2607.16206#bib.bib2)\]have demonstrated remarkable efficacy in stabilizing the training process and optimizing scalar reward signals\. Specifically, in the domain of reasoning, Reinforcement Learning from Verifiable Rewards \(RLVR\)\[[3](https://arxiv.org/html/2607.16206#bib.bib3)\]leverages objective feedback \(e\.g\., compiler outputs or mathematical correctness\) to steer models toward correct solutions\. However, these methods primarily focus on mode\-seeking behavior, rewarding the model for identifying any single path to a correct answer\. This often results in a "narrowing" of the policy, where the model sacrifices solution diversity for the sake of reward stability, a phenomenon we aim to mitigate\.

Exploration remains one of the most fundamental challenges in RL\. Traditional approaches typically rely on Intrinsic Motivation, such as curiosity\-driven rewards based on prediction errors\[[5](https://arxiv.org/html/2607.16206#bib.bib5)\], or Count\-based Exploration which penalizes frequently visited states\[[4](https://arxiv.org/html/2607.16206#bib.bib4)\]\. Furthermore, Entropy Regularization is widely employed to prevent premature convergence by maintaining a minimum level of stochasticity in the policy\[[6](https://arxiv.org/html/2607.16206#bib.bib6)\]\. While effective in low\-dimensional or discrete grid\-world environments, these techniques often falter in the high\-dimensional, sparse\-reward semantic space of LLMs\. Simple token\-level entropy or state\-counting fails to capture the structural or logical novelty of a reasoning chain, often leading to "pseudo\-exploration" where the model generates semantically identical but syntactically varied outputs\.

The concept of seeking "both good and different" solutions originates from Quality\-Diversity \(QD\) algorithms\[[7](https://arxiv.org/html/2607.16206#bib.bib7)\]and Novelty Search within the evolutionary computation community\[[8](https://arxiv.org/html/2607.16206#bib.bib8)\]\. Algorithms like MAP\-Elites\[[9](https://arxiv.org/html/2607.16206#bib.bib9)\]focus on illuminating the search space by maintaining a diverse population of high\-performing individuals\. While QD has seen success in robotics and procedural content generation, its integration into the gradient\-based optimization framework of modern Deep RL—especially for LLMs—remains non\-trivial\. PPO\-HSC bridges this gap by elegantly distilling the "valid\-yet\-diverse" philosophy of QD into a differentiable reward signal\. Unlike traditional QD which often relies on discrete archives, our approach utilizes a High\-order Sampling Coverage \(HSC\) mechanism to dynamically guide the policy toward unexplored regions of the reasoning manifold within the PPO framework\.

## 3Method

In this section, we elaborate on the exploratory reinforcement learning framework based on wide\-area policy coverage optimization \(PPO\-HSC\)\. We first review the standard reinforcement learning setup for large language models, then detail the construction mechanism of the dynamic valid trajectory library\. Based on this, we derive the core High\-kkSampling Coverage \(HSC\) reward function, and finally present the overall optimization objective and algorithm flow of PPO\-HSC\.

![Refer to caption](https://arxiv.org/html/2607.16206v1/PPO_HSC_Pipeline.png)Figure 2:PPO\-HSC: Wide\-Area Policy Coverage Optimization Framework\. The flowchart illustrates the iterative reinforcement learning loop of the PPO\-HSC framework: 1\. Initial Input: The process begins with the Policy Model \(Agent\) receiving a specific Prompt \(xx\)\. 2\. Trajectory Sampling: The framework performs M\-count sampling to generate a Target Sequence \(St​a​r​g​e​tS\_\{target\}\) alongside a Coverage Set \(CxC\_\{x\}\) of alternative trajectories\. 3\. High\-order Sampling Coverage \(HSC\) Module: This core component calculates the intrinsic reward by measuring similarity between the target sequence, the coverage set, and the Dynamic Trajectory Library \(𝒟\\mathcal\{D\}\)\. It includes Verifiability Filtering to check for correctness \(Ro​b​j=1R\_\{obj\}=1\) and a Plausibility Constraint \(KL\) relative to a Reference Model to ensure logical soundness\.4\. Final Reward Aggregation: The total reward \(Rt​o​t​a​lR\_\{total\}\) is aggregated from the HSC reward \(rh​s​cr\_\{hsc\}\), the KL penalty \(rk​lr\_\{kl\}\), and the objective/ground\-truth reward \(rg​tr\_\{gt\}\)\.5\. PPO Update Mechanism: The aggregated reward is used to calculate advantages via GAE to update the Policyπθ\\pi\_\{\\theta\}through gradient ascent\.Optimization Goal: This cycle repeats to achieve Wide\-Area Coverage and a Diversity Boost, breaking "mode collapse" by incentivizing the model to discover novel, valid reasoning paths### 3\.1Preliminaries

In the reinforcement learning fine\-tuning of Large Language Models \(LLMs\), the text generation process is typically modeled as a Markov Decision Process \(MDP\)\. Given an initial promptxx, the model needs to generate the next tokenat∈𝒱a\_\{t\}\\in\\mathcal\{V\}under the current context statests\_\{t\}, where𝒱\\mathcal\{V\}is the vocabulary\. The policyπθ​\(at\|st\)\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)is represented by the parameterized LLM\. In Reinforcement Learning from Verifiable Rewards \(RLVR\), after the model generates a complete trajectoryτ=\(a0,a1,…,aT\)\\tau=\(a\_\{0\},a\_\{1\},\\dots,a\_\{T\}\), the environment provides an objective verification rewardRo​b​j​\(τ\)R\_\{obj\}\(\\tau\)\. In code generation or mathematical reasoning tasks, this reward is usually an extremely sparse binary reward, i\.e\.:

Ro​b​j​\(τ\)=\{1,if​τ​passes objective verification \(e\.g\., unit tests or answer matching\)0,otherwiseR\_\{obj\}\(\\tau\)=\\begin\{cases\}1,&\\text\{if \}\\tau\\text\{ passes objective verification \(e\.g\., unit tests or answer matching\)\}\\\\ 0,&\\text\{otherwise\}\\end\{cases\}The traditional PPO algorithm aims to maximize the expected reward𝔼τ∼πθ​\[Ro​b​j​\(τ\)\]\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\[R\_\{obj\}\(\\tau\)\]\. However, when facing an extremely vast solution space, once the model accidentally discovers a valid trajectory \(Ro​b​j=1R\_\{obj\}=1\), the gradient update rapidly increases the probability of this trajectory\. This causes the model to fall into an "invisible constraint" \(i\.e\., mode collapse\), losing the ability to explore the expansive action space to discover better or more diverse solutions\.

### 3\.2Dynamic Valid Trajectory Library

To guide the policy to break out of known high\-reward paths, PPO\-HSC introduces a dynamically maintained memory module: the valid trajectory library𝒟\\mathcal\{D\}\. This library is specifically designed to record all verified correct unique trajectories explored by the base model under an extremely high number of sampling iterations \(High\-kk\) or by the current policy during training\. At training steptt, the trajectory library is defined as a finite set:

𝒟t=\{τ1,τ2,…,τN\}\\mathcal\{D\}\_\{t\}=\\\{\\tau\_\{1\},\\tau\_\{2\},\\dots,\\tau\_\{N\}\\\}The updating of the library follows a strict admission mechanism: a newly sampled trajectoryτn​e​w\\tau\_\{new\}is added to the library if and only if it satisfiesRo​b​j​\(τn​e​w\)=1R\_\{obj\}\(\\tau\_\{new\}\)=1and is not already contained in𝒟t\\mathcal\{D\}\_\{t\}at the semantic or symbolic level\. To prevent out\-of\-memory errors and maintain retrieval efficiency, when the library size exceeds a set thresholdNm​a​xN\_\{max\}, we employ a Diversity\-Greedy strategy\.

Specifically, we define a distance metricd​\(τi,τj\)d\(\\tau\_\{i\},\\tau\_\{j\}\)to measure the semantic or structural dissimilarity between any two valid trajectories\. When a newly verified trajectoryτn​e​w\\tau\_\{new\}arrives and the current library size\|𝒟t\|≥Nm​a​x\|\\mathcal\{D\}\_\{t\}\|\\geq N\_\{max\}, we construct a temporary expanded set𝒟′=𝒟t∪\{τn​e​w\}\\mathcal\{D\}^\{\\prime\}=\\mathcal\{D\}\_\{t\}\\cup\\\{\\tau\_\{new\}\\\}\. We then evaluate the marginal diversity contribution of each trajectory in𝒟′\\mathcal\{D\}^\{\\prime\}, typically computed as the distance to its nearest neighbor in the set\. The strategy iteratively identifies and evicts the trajectoryτe​v​i​c​t\\tau\_\{evict\}that contributes the least to the overall diversity of the library\. Consequently, the updated library is defined as𝒟t\+1=𝒟′∖\{τe​v​i​c​t\}\\mathcal\{D\}\_\{t\+1\}=\\mathcal\{D\}^\{\\prime\}\\setminus\\\{\\tau\_\{evict\}\\\}\. This eviction mechanism ensures that highly redundant paths are pruned while preserving a fixed\-size, maximally dispersed subset of high\-reward examples to continuously guide the policy toward novel solution spaces\.

### 3\.3High\-k Sampling Coverage Reward \(HSC\)

The High\-order Sampling Coverage \(HSC\) reward serves as the primary engine for intrinsic motivation within the PPO\-HSC framework\. Its core design philosophy is to move beyond simple reward maximization and instead incentivize "boundary expansion"\. Specifically, the mechanism rewards generated trajectories that exhibit significant semantic or structural novelty compared to historically known valid experiences, while simultaneously ensuring these paths remain within the model’s plausible reasoning manifold\. The HSC reward is mathematically structured as a composite of two mutually restricting terms: a Novelty Measure and a Plausibility Constraint\.

#### Novelty Measure

Novelty aims to calculate the distance between the current trajectoryτ\\tauand the most similar trajectory in the library𝒟t\\mathcal\{D\}\_\{t\}\. Letϕ​\(⋅\)\\phi\(\\cdot\)be the encoding function that maps discrete sequences to the average\-pooled representation of the base model’s final layer: Let a trajectoryτ\\taube defined as a sequence ofLLdiscrete tokens:τ=\(x1,x2,…,xL\)\\tau=\(x\_\{1\},x\_\{2\},\\dots,x\_\{L\}\)When this sequence is processed by the base model, the final layer outputs a corresponding sequence of dense hidden state vectors:H=\(h1,h2,…,hL\)H=\(h\_\{1\},h\_\{2\},\\dots,h\_\{L\}\)Here, eachhi∈ℝdh\_\{i\}\\in\\mathbb\{R\}^\{d\}represents the contextualized embedding of theii\-th token, andddis the hidden dimension of the model\.The average\-pooled representation,ϕ​\(τ\)\\phi\(\\tau\), is calculated by taking the arithmetic mean of these hidden vectors across the time \(or sequence\) dimension:

ϕ​\(τ\)=1L​∑i=1Lhi\\phi\(\\tau\)=\\frac\{1\}\{L\}\\sum\_\{i=1\}^\{L\}h\_\{i\}We define the minimum neighbor distance as:

dm​i​n​\(τ,𝒟t\)=minτi∈𝒟t⁡\(1−ϕ​\(τ\)⋅ϕ​\(τi\)‖ϕ​\(τ\)‖​‖ϕ​\(τi\)‖\)d\_\{min\}\(\\tau,\\mathcal\{D\}\_\{t\}\)=\\min\_\{\\tau\_\{i\}\\in\\mathcal\{D\}\_\{t\}\}\\left\(1\-\\frac\{\\phi\(\\tau\)\\cdot\\phi\(\\tau\_\{i\}\)\}\{\\\|\\phi\(\\tau\)\\\|\\\|\\phi\(\\tau\_\{i\}\)\\\|\}\\right\)In tasks with strong symbolic logic, the normalized Levenshtein Distance can also be used directly:

dm​i​n​\_​s​y​m​\(τ,𝒟t\)=minτi∈𝒟t⁡\(Levenshtein​\(τ,τi\)max⁡\(\|τ\|,\|τi\|\)\)d\_\{min\\\_sym\}\(\\tau,\\mathcal\{D\}\_\{t\}\)=\\min\_\{\\tau\_\{i\}\\in\\mathcal\{D\}\_\{t\}\}\\left\(\\frac\{\\text\{Levenshtein\}\(\\tau,\\tau\_\{i\}\)\}\{\\max\(\|\\tau\|,\|\\tau\_\{i\}\|\)\}\\right\)A larger distancedm​i​nd\_\{min\}indicates a higher "exploration gain" for the trajectory\.

#### Plausibility Constraint

A common failure mode in curiosity\-driven RL is "pseudo\-exploration," where the model generates high\-entropy gibberish just to maximize novelty\. To mitigate this, PPO\-HSC introduces a token\-level KL divergence penalty relative to a frozen reference base modelπr​e​f\\pi\_\{ref\}:

PK​L​\(τ\)=−∑t=0Tlog⁡πθ​\(at\|st\)πr​e​f​\(at\|st\)P\_\{KL\}\(\\tau\)=\-\\sum\_\{t=0\}^\{T\}\\log\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\}\{\\pi\_\{ref\}\(a\_\{t\}\|s\_\{t\}\)\}This constraint acts as a "sanity check," ensuring that while the model explores new areas, it does not deviate so far from the natural language manifold that it loses structural rationality or syntax integrity\.

#### Comprehensive HSC Reward Function

By synthesizing these terms, the final HSC reward for a single trajectory is defined as:

RH​S​C​\(τ,𝒟t\)=tanh⁡\(dm​i​n​\(τ,𝒟t\)σ\)\+β⋅PK​L​\(τ\)R\_\{HSC\}\(\\tau,\\mathcal\{D\}\_\{t\}\)=\\tanh\\left\(\\frac\{d\_\{min\}\(\\tau,\\mathcal\{D\}\_\{t\}\)\}\{\\sigma\}\\right\)\+\\beta\\cdot P\_\{KL\}\(\\tau\)Here,σ\\sigmais a temperature hyperparameter that adjusts the sensitivity to distance, whileβ\\betacontrols the weight of the plausibility constraint\. The use of thetanh\\tanhfunction ensures that the novelty reward is bounded, preventing any single outlier trajectory from dominating the gradient update and destabilizing the training process\. This balanced signal encourages the discovery of "low\-similarity yet high\-validity" reasoning patterns\.

### 3\.4Overall Optimization Objective of PPO\-HSC

The primary innovation of the PPO\-HSC framework lies in its specialized reward fusion mechanism, which systematically integrates task\-specific objectives with exploratory incentives\. Unlike traditional information entropy regularization, which may blindly encourage stochasticity or "pseudo\-exploration" \(generating semantically identical but syntactically varied outputs\), PPO\-HSC constrains its exploration pressure strictly to the manifold of objective rewards\. This ensures that the model is not merely rewarded for being "different," but for being "different and correct"\. We define the total aggregated reward for a sampled trajectoryτ\\tauas follows:

Rt​o​t​a​l​\(τ\)=Ro​b​j​\(τ\)\+α⋅𝕀​\[Ro​b​j​\(τ\)=1\]⋅RH​S​C​\(τ,𝒟t\)R\_\{total\}\(\\tau\)=R\_\{obj\}\(\\tau\)\+\\alpha\\cdot\\mathbb\{I\}\[R\_\{obj\}\(\\tau\)=1\]\\cdot R\_\{HSC\}\(\\tau,\\mathcal\{D\}\_\{t\}\)In this formulation,Ro​b​j​\(τ\)R\_\{obj\}\(\\tau\)represents the sparse binary reward derived from objective verification \(e\.g\., unit test results or mathematical answer matching\), and𝕀​\[⋅\]\\mathbb\{I\}\[\\cdot\]is an indicator function\. The inclusion of this indicator function is a critical design choice: it ensures that the model only receives the additionalRH​S​CR\_\{HSC\}"uniqueness" bonus if it has first successfully solved the problem\. By gating the exploration reward behind the verification of correctness, we prevent the policy from drifting into regions of the action space that produce novel but logically invalid or nonsensical reasoning chains\. The coefficientα\\alphaserves as the coverage reward weight, allowing for fine\-tuned control over the balance between exploiting known solutions and expanding the policy boundary\. Once the total rewardRt​o​t​a​l​\(τ\)R\_\{total\}\(\\tau\)is computed, the framework transitions to the policy update phase\. To stabilize training and mitigate the high variance often associated with sparse rewards in Large Language Models, we calculate the advantage functionA^t\\hat\{A\}\_\{t\}using Generalized Advantage Estimation \(GAE\)\. This advantage signal is then substituted into the standard PPO clipped surrogate objective function:

ℒC​L​I​P​\(θ\)=𝔼^t​\[min⁡\(rt​\(θ\)​A^t,clip​\(rt​\(θ\),1−ϵ,1\+ϵ\)​A^t\)\]\\mathcal\{L\}^\{CLIP\}\(\\theta\)=\\hat\{\\mathbb\{E\}\}\_\{t\}\\left\[\\min\(r\_\{t\}\(\\theta\)\\hat\{A\}\_\{t\},\\text\{clip\}\(r\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\_\{t\}\)\\right\]wherert​\(θ\)=πθ​\(at\|st\)πo​l​d​\(at\|st\)r\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\}\{\\pi\_\{old\}\(a\_\{t\}\|s\_\{t\}\)\}denotes the probability ratio between the current and old policy, andϵ\\epsilonis the clipping threshold used to prevent destructively large policy updates\. Simultaneously, the value network \(Critic\), parameterized byϕ\\phi, is updated to provide more accurate baseline estimates by minimizing the mean squared error \(MSE\) relative to the empirical returnsR¯t\\overline\{R\}\_\{t\}:

ℒV​F=‖Vϕ​\(st\)−R¯t‖2\\mathcal\{L\}^\{VF\}=\\\|V\_\{\\phi\}\(s\_\{t\}\)\-\\overline\{R\}\_\{t\}\\\|^\{2\}Through this synchronous update of the policy and value networks, PPO\-HSC effectively transforms the "Invisible Shackles" of mode collapse into a structured search for wide\-area coverage\. This iterative process incentivizes the model to discover diverse reasoning paths while maintaining the strict syntax and logical integrity required for complex reasoning and code generation tasks\.

The training procedure of PPO\-HSC is outlined in algorithm 1\.

Algorithm 1PPO\-HSC Policy Optimization0:Initial policy

πθ\\pi\_\{\\theta\}, reference model

πref\\pi\_\{\\text\{ref\}\}, value network

VϕV\_\{\\phi\}
0:Prompt distribution

𝒟\\mathcal\{D\}, objective reward function

RobjR\_\{\\text\{obj\}\}
0:Hyperparameters: HSC weight

α\\alpha, KL penalty weight

β\\beta, temperature

σ\\sigma, clipping ratio

ϵ\\epsilon
1:Initialize trajectory library

𝒟0←∅\\mathcal\{D\}\_\{0\}\\leftarrow\\emptyset
2:

πold←πθ\\pi\_\{\\text\{old\}\}\\leftarrow\\pi\_\{\\theta\}
3:whilenot convergeddo

4:Sample a prompt

q∼𝒟q\\sim\\mathcal\{D\}
5:Generate a trajectory

τ=\(a0,…,aT\)∼πold\(⋅∣q\)\\tau=\(a\_\{0\},\\dots,a\_\{T\}\)\\sim\\pi\_\{\\text\{old\}\}\(\\cdot\\mid q\)
6:Evaluate objective reward

Robj​\(τ\)R\_\{\\text\{obj\}\}\(\\tau\)
7:if

Robj\(τ\)==1R\_\{\\text\{obj\}\}\(\\tau\)==1then

8:Compute minimum distance

dmin​\(τ,𝒟k−1\)d\_\{\\min\}\(\\tau,\\mathcal\{D\}\_\{k\-1\}\)to the existing library

9:Compute plausibility constraint

PKL​\(τ\)=−∑t=0Tlog⁡πθ​\(at∣st\)πref​\(at∣st\)P\_\{\\text\{KL\}\}\(\\tau\)=\-\\sum\_\{t=0\}^\{T\}\\log\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\text\{ref\}\}\(a\_\{t\}\\mid s\_\{t\}\)\}
10:Compute HSC reward

RHSC←tanh⁡\(dminσ\)\+β⋅PKL​\(τ\)R\_\{\\text\{HSC\}\}\\leftarrow\\tanh\\left\(\\frac\{d\_\{\\min\}\}\{\\sigma\}\\right\)\+\\beta\\cdot P\_\{\\text\{KL\}\}\(\\tau\)
11:Update trajectory library

𝒟k←𝒟k−1∪\{τ\}\\mathcal\{D\}\_\{k\}\\leftarrow\\mathcal\{D\}\_\{k\-1\}\\cup\\\{\\tau\\\}
12:else

13:

RHSC←0R\_\{\\text\{HSC\}\}\\leftarrow 0
14:

𝒟k←𝒟k−1\\mathcal\{D\}\_\{k\}\\leftarrow\\mathcal\{D\}\_\{k\-1\}
15:endif

16:Compute total reward

Rtotal​\(τ\)=Robj​\(τ\)\+α⋅RHSCR\_\{\\text\{total\}\}\(\\tau\)=R\_\{\\text\{obj\}\}\(\\tau\)\+\\alpha\\cdot R\_\{\\text\{HSC\}\}
17:Compute advantage estimates

A^t\\hat\{A\}\_\{t\}and returns

R^t\\hat\{R\}\_\{t\}using Generalized Advantage Estimation \(GAE\)

18:forgradient update iterationsdo

19:Compute probability ratio

rt​\(θ\)=πθ​\(at∣st\)πold​\(at∣st\)r\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\text\{old\}\}\(a\_\{t\}\\mid s\_\{t\}\)\}
20:Compute PPO\-HSC objective

ℒCLIP​\(θ\)\\mathcal\{L\}^\{\\text\{CLIP\}\}\(\\theta\)using

rt​\(θ\)r\_\{t\}\(\\theta\),

A^t\\hat\{A\}\_\{t\}and

ϵ\\epsilon
21:Update policy

πθ\\pi\_\{\\theta\}via gradient ascent on

ℒCLIP​\(θ\)\\mathcal\{L\}^\{\\text\{CLIP\}\}\(\\theta\)
22:Update value network

VϕV\_\{\\phi\}via gradient descent on MSE loss

‖Vϕ​\(st\)−R^t‖2\\\|V\_\{\\phi\}\(s\_\{t\}\)\-\\hat\{R\}\_\{t\}\\\|^\{2\}
23:endfor

24:

πold←πθ\\pi\_\{\\text\{old\}\}\\leftarrow\\pi\_\{\\theta\}
25:endwhile

26:returnOptimized policy

πθ\\pi\_\{\\theta\}

## 4Experiments

### 4\.1Experimental Setup

#### Datasets

We evaluate PPO\-HSC on two distinct domains: mathematical reasoning and code generation\. For mathematics, we utilize GSM8K and SVAMP, which require multi\-step logical deduction\. For code generation, we evaluate the model’s ability to produce functional Python segments using the same datasets processed through a code\-execution verifier\.

#### Baselines

Base Model:Qwen2\.5\-7B\-Instruct without further reinforcement learning\.PPO\-only:A standard PPO fine\-tuning baseline using only binary objective rewards \(Ro​b​jR\_\{obj\}\), representing the common Reinforcement Learning from Verifiable Rewards \(RLVR\) paradigm\.

#### Implementation Details

We set the sampling sizek=5k=5for each prompt during training\. The High\-order Sampling Coverage \(HSC\) reward incorporates a novelty measuredm​i​nd\_\{min\}based on average\-pooled representations and aPK​LP\_\{KL\}plausibility constraint\. We utilize LoRA for parameter\-efficient fine\-tuning to balance performance and computational cost\.

### 4\.2Main Results on Mathematical Reasoning

The results for mathematical reasoning are summarized in Table[1](https://arxiv.org/html/2607.16206#S4.T1)\. PPO\-HSC demonstrates a clear advantage in discovering multiple valid reasoning paths\.

Table 1:Comparison of mathematical reasoning performance\. Results for PPO\-HSC are reported at the best\-performing checkpoint \(†\\dagger\)\.#### Superior Multi\-path Reasoning

As shown in Table[1](https://arxiv.org/html/2607.16206#S4.T1), PPO\-HSC achieves a Pass@5 of 0\.405 on GSM8K and 0\.685 on SVAMP, outperforming the Base model and PPO\-only\. While standard PPO often suffers from performance degradation in later stages \(e\.g\., SVAMP Pass@5 dropping to 0\.650\), PPO\-HSC maintains superior performance\. The increase in Self\-Consistency \(SC@5\) from 0\.315 to 0\.320 on GSM8K further proves that our method encourages the model to generate a more diverse set of correct trajectories\.

### 4\.3Results on Code Generation and Syntax Stability

For code generation, maintaining syntax correctness while exploring the policy space is a significant challenge\. Table[2](https://arxiv.org/html/2607.16206#S4.T2)highlights how PPO\-HSC addresses this\.

Table 2:Code generation performance and syntax integrity analysis on the SVAMP dataset\.#### Robustness Against Mode Collapse

Standard PPO \(PPO\-only\) exhibits a decline in Pass@1 \(0\.560 vs Base’s 0\.580\) on SVAMP, indicating a collapse toward sub\-optimal or repetitive code structures\. PPO\-HSC not only reverses this trend but also achieves the highest Pass@5 \(0\.685\)\.

#### Preserving Syntax

Crucially, PPO\-HSC maintains a 1\.000 Syntax@5 rate, matching the Base model\. This indicates that our Plausibility Constraint effectively filters out low\-quality or syntactically invalid exploratory trajectories, ensuring that the "Wide\-Area Coverage" optimization occurs only within the manifold of valid code\.

![Refer to caption](https://arxiv.org/html/2607.16206v1/charts.png)Figure 3:A 2×2 tianzige\-style visualization comparing Base, PPO\-only, and PPO\-HSC \(Ours\)\. Each panel uses blue\-gradient bar charts \(dark to light\) to present key metrics across GSM8K, SVAMP, syntax scores, and overall mean performance\. The consistent layout and color scheme make it easy to compare model behavior across datasets and evaluation dimensions\.We analyze the Unique Code Ratio to evaluate exploration efficiency\. In SVAMP tasks, PPO\-HSC maintains an avg\_unique\_code\_ratio@5 of approximately 0\.333, similar to the base model, even as its accuracy improves\. This demonstrates that PPO\-HSC successfully expands the policy boundary to discover new, correct reasoning paths without sacrificing the diversity inherent in the pre\-trained model\.

## 5Conclusion

In this paper, we introduced PPO\-HSC \(Proximal Policy Optimization with High\-order Sampling Coverage\), a novel reinforcement learning framework designed to break the "Invisible Shackles" of mode collapse in Large Language Model fine\-tuning\. By shifting the optimization target from singular path exploitation to wide\-area sampling space coverage, PPO\-HSC encourages models to explore diverse, logically sound reasoning manifolds\.Our core technical contribution is the High\-order Sampling Coverage \(HSC\) reward mechanism\. By maintaining a dynamic trajectory library of unique, verified solutions, the framework provides an intrinsic reward for trajectories that are semantically novel yet structurally rational\. This mechanism, coupled with a plausibility constraint based on KL divergence, ensures that exploration remains within the manifold of valid solutions without descending into "pseudo\-exploration" or meaningless gibberish\.

Empirical results across mathematical reasoning and code generation tasks demonstrate the efficacy of our approach:

- •Enhanced Diversity:PPO\-HSC significantly improves Pass@5 and Self\-Consistency metrics, outperforming standard PPO baselines by discovering multiple valid reasoning paths\.
- •Robustness to Mode Collapse:Unlike traditional PPO, which often suffers from performance degradation and repetitive structures, PPO\-HSC maintains superior accuracy and a healthy unique code ratio\.
- •Syntax Integrity:The framework preserves a 1\.000 Syntax@5 rate in code generation, proving that wide\-area coverage can be achieved without sacrificing the structural requirements of the task\.

By elegantly distilling the philosophy of Quality\-Diversity algorithms into a differentiable reward signal, PPO\-HSC provides a robust path forward for developing LLMs that are not just "accelerators of known solutions," but creative and versatile reasoners\. Future work will focus on scaling this framework to even higher\-dimensional semantic spaces and exploring its application in open\-ended creative tasks\.

## References

- \[1\]Schulman, J\., et al\.: Proximal Policy Optimization Algorithms\. arXiv:1707\.06347 \(2017\)
- \[2\]Rafailov, R\., et al\.: Direct Preference Optimization: Your Language Model is Secretly a Reward Model\. Advances in Neural Information Processing Systems, 36 \(2023\)
- \[3\]Wen, X\., et al\.: Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs\. arXiv:2506\.14245 \(2025\)
- \[4\]Bellemare, M\., et al\.: Unifying Count\-Based Exploration and Intrinsic Motivation\. Advances in Neural Information Processing Systems, 29 \(2016\)
- \[5\]Pathak, D\., et al\.: Curiosity\-Driven Exploration by Self\-Supervised Prediction\. In: International Conference on Machine Learning, pp\. 2771\-2780\. PMLR \(2017\)
- \[6\]Williams, R\. J\., Peng, J\.: Function Optimization Using Connectionist Reinforcement Learning Algorithms\. Connection Science, 3\(3\), 241\-268 \(1991\)
- \[7\]Pugh, J\. K\., et al\.: Quality Diversity: A New Frontier for Evolutionary Computation\. Frontiers in Robotics and AI, 3, 40 \(2016\)
- \[8\]Lehman, J\., Stanley, K\. O\.: Abandoning Objectives: Evolution Through the Search for Novelty Alone\. Evolutionary Computation, 19\(2\), 189\-223 \(2011\)
- \[9\]Mouret, J\. B\., Clune, J\.: Illuminating Search Spaces by Mapping Elites\. arXiv:1504\.04909 \(2015\)