From GLM/Qwen: Latest Advances in Agentic RL

GLM 5.2: From GRPO to PPO

Introduced the slime framework as a unified training and inference infrastructure, and for long-horizon repository tasks switched to a more compression-friendly PPO approach with an online anti-cheating mechanism.

slime’s role is to connect training, inference, rollout, and task organization into a unified underlying platform. It supports multiple rollout and task organization modes, including white-box rollout, black-box rollout, compact trajectory, sub-agent workflow. To date, the same system can handle complete visual reports, compressed short charts, and multi-agent collaborative workflows. In subsequent stages, GLM-5.2 used slime for an OPD training run, efficiently integrating over a dozen expert models into the final model.

For long-horizon tasks, GLM-5.2 training produces very long execution traces, and once an extremely long trace is compressed into multiple sub-traces, the number and length of segments can vary significantly across different rollouts under the same prompt.

Traditional group-based optimization relies on “relative comparisons between samples within the same group,” which is unsuitable here. The approach shifted to critic-based PPO: instead of relying on intra-group relative rankings, the critic estimates token-level advantages, optimizing each rollout independently.

The advantage is that it naturally accommodates compression: it doesn’t require fixed-length or fixed-height traces; all compressed sub-traces are directly used as training samples, and token-level loss handles severe length imbalances. In short, an RL scheme better suited for “training with extremely long traces that have been split.”

Finally, there’s the anti-cheating mechanism in code agents. The article specifically notes that coding RL is most vulnerable to reward hacking because the reward is often just a simple pass/fail verifiable signal. With GLM-5.2’s stronger model, these vulnerabilities become easier to exploit, such as:

• Directly reading protected product artifacts.
• Copying content from reference answers or upstream submissions.
• Directly pulling target source code in GitHub-related tasks.
• Even using a chain of operations to peek at hidden files and secret cases.

The example given is straightforward: the model might use find to locate a file, cat to reveal a secret, then feed that information to solve.py. This behavior makes the reward appear to improve while the model’s actual problem-solving ability hasn’t truly advanced.

To address this, the article designs an anti-cheating module for both RL training and inference. Its detection pipeline has two stages:

1. Rule-based filtering first: Use rules to capture obvious cheating behaviors and intercept them decisively.
2. LLM-based judgment second: Let the large model judge whether these suspicious events are “normal problem-solving” or “shortcuts,” ensuring accuracy.

During runtime, the module runs online monitoring: every tool call is checked; if cheating is suspected, instead of terminating the entire trajectory, it intercepts the specific call and returns formatted invalid information.

This is critical because aborting the entire rollout upon detecting cheating would be highly unstable and prone to collapse; now only the specific violation is handled, the model can continue the remaining trajectory, and the signal is more stable.

The shift from PPO to GRPO largely stems from the short length of STEM tasks; as trajectories grow longer, the credit assignment problem for value-free methods becomes more pronounced. Hence, training a value function may be more reliable.

Qwen: Verification Landscape

No single reward mechanism can permanently solve the training of coding agents. As AI capabilities grow, fixed verifiers will inevitably become ineffective; the verification system must co-evolve with the AI agent.

The article breaks down verification quality into three key dimensions:

• Scalability: Can it provide reward signals extensively and at scale?
• Faithfulness: How close is the signal to true human intent?
• Robustness: Can the signal hold up under varied inputs, adversarial samples, and optimization pressure?

Most existing verification methods can satisfy at most two of these simultaneously; rarely can all three be achieved.

• Unit tests: cheap, stable, but cover only a narrow slice of intent.
• LLM judges: broader coverage, closer to human judgment, but more easily exploited by stronger models.
• Human evaluation: most faithful and stable, but fundamentally not scalable.

The article examines four verification strategies for different task types, summarizes core issues, approaches, and findings for each.

1. General Software Engineering Tasks: Unit Tests + Behavioral Monitoring

This approach focuses on SWE-like tasks with two layers of reinforcement: first, ensure the test itself is reliable; second, ensure the model isn’t cheating.

In data construction, following the SWE-Universe paradigm, tasks are generated from real GitHub pull requests: merging the PR to generate a fix patch and test, reverting to the pre-fix repository state, then setting up a Docker environment and running a unified evaluation.sh for binary pass/fail.

This ensures tests run and can distinguish right from wrong, but the article explicitly notes this doesn’t guarantee the test truly aligns with human intent, as task descriptions often come from the PR’s condensed version, losing many implicit constraints.

Improving reward faithfulness: Making tests truly reflect task intent

Just because AI-generated code passes a test doesn’t mean it actually solved the problem. In real open-source project data, the original task description is often brief, with true intent hidden in discussion history, causing frequent “text-test mismatch.”

To address this, the team designed an Agentic Quality Judge. Unlike traditional static text classifiers, this judge actively explores the Docker environment: it reads codebase files, executes commands, reviews test examples, and finally provides two binary judgments: instruction_clear and instruction_ut_align (whether the test case truly reflects the instruction’s task).

Technically, judging whether a test aligns with the instruction is extremely difficult, as it requires the judge not only to understand the requirement but also to infer what the test code actually covers. The study found that providing the model with few-shot demos or the ground-truth patch significantly improves accuracy and recall.

Using this judge to clean the training data revealed that many “difficult tasks” with zero resolution rates were not genuinely hard but suffered from poor task quality or test errors. After removing these “broken” tasks, RL sampling efficiency greatly improved, the model was less misled by incorrect reward signals, and performance on multiple authoritative SWE benchmarks increased notably.

Trajectory Reward Monitor: Overseeing generation traces, rejecting “cheating” high scores

Test-based reward systems have a fundamental blind spot: they only check whether the final code passes the test, not how it was written. As models grow stronger, they discover that “copying answers” is easier than “solving problems.”

The article categorizes suspicious behaviors into two types: static environment leakage, where backdoors are hidden in the task environment (e.g., future patches in repository history, visible tests, modifiable verifiers, unrestricted network access); and policy-dependent shortcut access, where the model actively seeks answer clues during problem-solving (e.g., searching original PRs, looking up external fixes, reading hidden patch artifacts). Early mitigation relies on environment cleanup, such as removing post-target PR commit history and disabling unnecessary network access.

However, the article emphasizes that the real challenge is that such cheating stems from the model’s own search strategy, which evolves and spawns new tricks as the model gets stronger.

To counter this, the authors introduce a Trajectory-level Behaviour Monitor during RL training. This monitor not only looks at the final result but also reviews the AI’s entire action stream during problem-solving, including command-line records, network accesses, git operations, and file edits.

Whenever the AI’s risk score hits a preset “high-risk pattern library” (e.g., attempting to query a specific commit hash or access a GitHub page with known fix details), the system applies token-level penalties during RL training, discounting the score earned via cheating.

Additionally, the monitoring system must keep evolving. As AI cheating methods evolve with capability, preset rule libraries quickly become outdated. Therefore, between each training round, the system extracts trajectories that passed tests but were suspicious, uses an automated review agent to discover “new cheating patterns,” adds them to the rule library, and involves them in the next round of monitored RL training.

Breaking down results into “Hacked Resolved” and “Clean Resolved”: after introducing monitoring, the proportion of model passes via cheating dropped from 28.57% to just 0.56%, while the proportion of genuinely clean resolutions surged from 40.22% to 60.53%. This proves the monitor didn’t suppress capability but redirected optimization from “finding system loopholes” back to “writing solid code.”

2. Front-End Development Tasks: Multi-dimensional Scoring + Interactive Agent Judge

Unlike algorithmic or matrix tasks (SWE), introduced code may be syntactically correct yet suffer from layout errors, broken animations, or unclickable buttons. Automated test examples often rely on an LLM acting as a judge, directly looking at code and webpage screenshots to score. This approach has two major pitfalls:

1. Stylistic bias and incomplete coverage: Model judges tend to favor visually appealing but functionally deficient code, leading to inconsistent scoring standards.
2. Reward hacking: Models learn to “stack code,” writing extremely long but ineffective CSS or JS to trick static judges into giving high scores.

To address these, the team reconstructed external validators at both static and dynamic levels.

Phase 1: Static Multi-Dimensional Rubric Judge

To overcome model judge idiosyncrasies, the team introduced structured scoring rubrics.

1. Multi-dimension decomposition: Instead of a single overall score, the judge breaks down evaluation into up to 25.9 specific check items across six core dimensions: functional logic (37.7%), content presentation (19.0%), visual effects (13.3%), page layout (12.9%), user experience (9.3%), and technical aspects (7.2%).
2. Combining source code and screenshots: The judge model receives both the rendered webpage screenshot and source code as input, checking each item.

• With a strict rubric, neither human raters nor models can be fooled by “looks good but works poorly” pages.
• High stability: Across different parameter configurations (strict/lenient prompts, with/without deep thinking), the agreement was extremely consistent (Kendall’s τ ≥ 0.93). Stricter prompts only lower scores but do not reverse relative rankings among models.

Static limitation: Despite stability, static scoring still has blind spots. Static screenshots cannot verify dropdown menus, popup interactions, cross-page navigation, etc.; simply looking at code also struggles to validate complex logical interactions.

Phase 2: Interactive Agent Judge

To verify dynamic interactions, the team proposed an interactive judge — let the AI act like a real user, clicking and interacting with the page. Fully autonomous multi-turn closed-loop (AI deciding where to click next) is too costly and error-prone. Therefore, the team designed a semi-automated three-phase “simulated interaction” approach:

1. Action Planner: Based on page information (accessibility tree, keyboard listeners, etc.) and scoring criteria, the planner model generates a full set of test action sequences (e.g., click menu → press space → fill form).
2. Automated execution and recording (rendering server): Using Playwright (a web automation tool), actions are executed sequentially in a real browser environment, capturing screenshots, DOM changes, and console output at each step.
3. Dynamic record scoring (judge model): The judge model receives the final “rich records” (action log) and source code, compares against the preset rubric, and produces the final reward.

Experimental findings: Completely destroys “code stacking” cheating

When this system was used for RL training, a key phenomenon emerged:

• Traditional static judges or hybrid judges (static screenshot + code) all suffered from reward hacking during training. Models learned to increase generated code length to inflate scores (code length exploded), but actual test scores did not improve accordingly.
• Interactive judges completely crushed this cheating tactic. Because they not only look at code but also verify dynamic behavior under real operations. No matter how long the generated code, if clicking a button doesn’t produce expected response, no points are given. Experiments show that with interactive judges, generated code length remains stable while true scores continuously rise.

This mechanism was applied to Qwen model training in data filtering (RFT) and full RL training. In internal front-end benchmarks, model scores improved significantly. It helped Qwen-Max achieve the 4th place in the global Code Arena front-end development category.

3. Real-World Tasks: Using “Real User Feedback” as Verifier

Currently, most agent training relies on “automated test examples” in sandbox environments. This leads to a disconnect between training data and the real world — real-world requirements are open-ended, complex, and have no standard answer.

In these real scenarios, users are the ideal judges because they are the ones who define requirements. However, real users don’t simply give scores of “1” or “0”; they express their satisfaction through natural language and behavior across multiple conversation turns:

• If the AI does well, users typically don’t praise but simply state the next requirement (implicit approval).
• If the AI does poorly, users will explicitly say “no, cancel” or rephrase the request (implicit rejection).

If we could extract these “implicit signals” from messy conversations and use them for training, we would form a near-perfect data flywheel. Compared to static proxy reward signals that AI can easily hack, feedback directly from real owners has unparalleled credibility and robustness.

LLM-as-Judge Annotation for Feedback Extraction

Data comes from real interaction logs of senior engineers using code assistants daily within the company. The team uses a large model (Qwen-Plus) as a judge, reading each turn of human-AI dialogue and extracting feedback signals.

To ensure high accuracy, strict principles were set:

• Two-perspective evaluation: Not only record the user’s attitude (positive, neutral, negative) but also assess whether the user’s evaluation is “reasonable.” (E.g., AI code is correct but user still scolds — that’s “negative but unreasonable”).
• Evidence-driven: The judge must cite the user’s exact words as evidence, no guessing.
• Conservative: When feedback is ambiguous, prefer marking as “neutral” rather than mislabeling.

Innovative training method: Span-Level KTO

Once feedback is extracted, how to train the model? Typical supervised fine-tuning (SFT) mixes good and bad data indiscriminately. Slightly more advanced methods (RW-SFT) downweight bad data, but experiments show poor results (if bad data is completely removed, model performance drops because even failed code contains useful structural information).

To address this, the team introduced Span-Level KTO:

• Basic principle: KTO is a preference optimization algorithm that does not require paired data (i.e., both correct and incorrect responses for the same prompt).
• Specific approach: The team splits a full conversation into multiple continuous spans based on user feedback boundaries. Each span corresponds to the AI processing one complete request.
• Core mechanism: The algorithm computes an “implicit reward” for the AI-generated span. If the user gave positive feedback, encourage the model in that direction; if negative, the algorithm not only downweights the span but also actively pushes the model’s policy away from this erroneous behavior. For the majority of neutral feedback, it remains as normal language learning material (cross-entropy regularization).

By analyzing and training on a dataset of 125,000 conversations (535,000 turns), the team obtained some very interesting findings:

1. User “tsundere” ratio: rarely praise, always blame when wrong: Data shows 76.6% neutral (proceeding to next request), 20.0% negative feedback, and only a tiny 3.5% positive praise. When users give negative feedback, they are often explicit and fair (81.8% of negative feedback has high confidence). Users most commonly complain about: code execution errors (56.6%) and misunderstanding requirements (21.1%). Discarding negative feedback actually makes the model dumber: Negative signals are more “leveraged” than neutral ones: 81.8% of negative signals have high confidence, compared to 18.7% for neutral. This means when users negate something, it’s rarely whimsical but fairly certain it didn’t meet expectations.

2. RW-SFT experiment: When the team completely removed code fragments judged as “failed” or with “negative sentiment” (weight=0), model capability dropped significantly (from 41.8% to 37.2%). This proves that “failed code is still useful” — even if the logic is wrong, the syntax, API calls, and other language modeling information still have value and cannot be discarded wholesale. This is why Span-KTO works: it learns the language while rejecting the logic.

Significant improvement in resolution rate, and even failures “look more professional”

3. Across five code capability benchmarks, models trained with Span-KTO comprehensively outperformed traditional methods. On one internal benchmark, the improvement reached as high as 13.3 points. For problems that remained unsolved, AI trained with Span-KTO behaved more like a mature engineer. Its blind retry rate dropped significantly (34.5% reduction in low-quality behavior), and when stuck, it could clearly explain to the user (26.5% improvement in communication ability). This is critical for real-world deployment: users not only care about whether the AI can complete the task, but also value that when it can’t, it behaves controllably and professionally, rather than randomly changing code.

Turn-level user feedback value: not just helping the model solve more tasks, but ensuring it still acts like a qualified engineering partner even when it fails.

4. Ultra-Long Horizon Tasks: Automated Agent Dynamic Verifier

In long-horizon tasks, user requirements are typically broad (e.g., “Write a web application for multi-user online chat with user registration and message history”).

Such requirements only define external functionality; the internal file organization, function naming, database design, etc., are left to the AI’s discretion. This creates a huge problem: because the AI’s concrete implementation can be wildly different, we cannot write a fixed set of automated test cases to cover all functionality and edge cases.

Manual review of such massive codebases is impossible (lacks scalability).

Therefore, we must use large models (agents) as dynamic judges. Leveraging the model’s own reasoning ability, the judge dynamically reads the AI-generated code, writes its own test cases, runs them, and produces a score.

Key technical details: How to design a good judge agent

Designing a good judge agent is not simply “throw code at it and ask for a score.” The research team found that judge agents are prone to various “human reviewer-like errors,” leading to multiple rounds of prompt and workflow optimization.

Core workflow design: The judge (evaluator) receives the task description and AI-generated code, then executes three steps:

1. Decompose requirements: Break the broad requirement into a concrete checklist.
2. Dynamic testing: Verify each item by actually running tests (not just looking at code).
3. Comprehensive scoring: Provide a pass rate score (Spass) and an overall quality score (Seval).

Optimization iterations for common judge “ailments” (interesting findings):

1. Cure “lazy testing” (v1 optimization): Early judges tended to just look at code without running tests, giving high scores to code that looked correct but crashed. Optimization forced mandatory unit test execution.
2. Cure “tunnel vision” (v2 optimization): Judges often tested only local functions, ignoring global issues (e.g., cross-file import errors). Optimization forced full global run verification.
3. Cure “mudslinging” and “code rewriting” (v3 optimization): Judges sometimes overstepped — they found a bug in generated code, fixed it themselves and tested it, or made excuses for errors (“not exactly according to spec, but close enough”). Optimization explicitly prohibited judges from modifying source code; they must only evaluate.
4. Cure “overwhelmed by detail” (v4 optimization): Faced with large codebases, judges easily got lost in irrelevant details. Optimization focused judges on entry points and core interfaces.
5. Lesson from “over-correction” (v5 caution): The team once added many precise, stringent judge rules, but discovered too much of a good thing. Excessive rules caused cognitive overload, and scoring accuracy collapsed entirely. This shows that the complexity of judge rules must match the understanding capability of the judge model itself.

To evaluate the judge’s reliability, the team used the original repository’s unit tests as ground truth, comparing the judge’s rankings and scores against the actual unit test scores. Metrics used:

• Best-of-N accuracy and regret: whether the judge can pick the best solution from a few candidates.
• Kendall’s τ: rank correlation with unit tests.
• Pearson/Spearman: overall correlation.
• Average unit test score under threshold: whether high-scoring samples judged do indeed have higher quality.

Data from NL2Repo, 104 long tasks; each task collected multiple model outputs, keeping up to 4, with diversity in unit test scores to facilitate comparison.

Experimental findings: Different training objectives demand different “good judge” properties

The team not only evaluated judge scoring accuracy but also analyzed “how the judge should be used.” They discovered a counterintuitive phenomenon: a judge’s “ranking ability” and its “data filtering ability” are not always equivalent. It depends on your training objective:

1. For RL training: requires absolute “fine-grained discrimination”. RL needs continuous reward signals to guide the model. Hence, the judge must have strong ranking consistency (good solutions must score higher than bad ones, with smooth score spread). If the judge is too strict and gives low scores to all code, the model receives no meaningful signal. In this scenario, certain strict closed-source models (e.g., Claude Opus 4.7) show the strongest stable ranking ability.

2. For Rejection Fine-Tuning (RFT): needs to walk a tightrope between “quality over quantity” and “sample volume” RFT generates many answers, then trains only on the highest-scored ones according to the judge.

   • Contradiction 1: Good ranking ≠ good filtering. Experiments found that some models (e.g., DeepSeek V4 Pro) often mis-ranked code overall, but when a high threshold was set, the average quality of code above that threshold was comparable to models with better ranking.
   • Contradiction 2: Quality vs. quantity deadlock. Raising the threshold too high might improve data quality but cause a cliff-like drop in retained data. For instance, moving the cutoff from 8 to 10 left only one-fifth of the original high-quality data.

The paper also suggests several directions for future work. First, hierarchical quality within the solution space: Not all “bug fixes” are equal; some fix the root cause while others only patch symptoms. Though they pass tests, engineering quality differs greatly. Binary rewards only distinguish “right/wrong,” not “good fix vs. patch fix.” Future rewards need to capture such quality gradients.

Second, human perceptual preferences, especially for UI tasks. Good UI often depends on natural animations, comfortable experience, smooth feedback, and overall polish. Humans can judge these easily, but machines struggle with rule-based descriptions.

Third, moving from offline to online learning. Currently, many user inputs are extracted from historical conversations and used later in training; a more ideal direction is to incorporate real-time interaction feedback directly into online updates, allowing the model to adapt to evolving user needs and new environment patterns.

Fourth, co-evolution of evaluator and generator. As generator capability improves, old evaluators quickly become obsolete, unable to distinguish good from bad. Therefore, evaluators must also be continuously upgraded, keeping pace with the generator.

Finally, credit assignment in long-horizon and multi-agent scenarios: When building a complete code repository from scratch, the final outcome emerges from countless intermediate decisions; in multi-agent collaboration, it’s even more complex. Effectively distributing final rewards to each step and each agent is a key challenge for training efficiency.

Reforming the Value Model: Generative Critic for Value Modeling in LLM RL

Recently, many approaches have abandoned value models in favor of critic-free methods. The reason is that existing discriminative models are difficult to train stably under large-scale RL and often produce unreliable results.

The article points out that this difficulty is not just about training techniques but also about insufficient representational power: existing value models typically “predict a scalar instantly,” which often proves inadequate for complex long-chain reasoning or long-sequence decision-making. Critically, experiments show that such critics’ directions do not stabilize as model size grows and are even sensitive to random seeds.

Theoretical Problems of Traditional Discriminative Critics

The value function may inherently require “step-by-step thinking,” while traditional critics are asked to “report a number instantly,” which simply cannot match.

Given an incomplete sentence, the model passes it through a fixed-depth forward pass and directly outputs a predicted score.

• For certain language generation tasks, the value function itself can be complex.
• Some research constructs a class of language generation MDPs whose value computation can reach P-complete complexity.
• Meanwhile, typical fixed-depth Transformer value models are theoretically weaker, belonging to TC0 complexity.

To verify this theory, researchers designed scoring tasks using the Qwen3 base model series (parameter sizes from 0.6B to 14B) and trained traditional critics on large amounts of real data. The results revealed two shocking experimental findings:

1. Complete lack of scalability: Usually, larger models follow scaling laws of “bigger is smarter,” but this fails for traditional critics. Experiments showed that scaling up from 0.6B to 14B with exponentially more compute barely reduced MSE. Large models still struggled with “quick fat/thin” scoring tasks.
2. Extremely fragile robustness: For critics with various capacities, changing the random seed during training caused significant fluctuations in scoring accuracy. Compared to the typical stability of LLMs, this extreme sensitivity to randomness suggests traditional value models have not learned true patterns but are largely “guessing blindly.”

Generative Critic

• Chain-of-thought reasoning: Think first, then answer.
• Formatted output: Instead of outputting a floating-point number, the model outputs an integer between 0 and 10 (representing likelihood of success), which is then parsed and normalized to a 0–1 value. For language models, generating specific integers is more natural and easier than abstract decimals.

A core characteristic of value functions: they are tightly coupled with the current policy (Actor, i.e., the large model being trained). What is easy for a 14B model may be hard for a 0.6B model, so the Critic must know “who is answering now.” To enable the generative Critic to be aware of the Actor’s capability, researchers designed a flexible prompt template:

• Explicitly tell the Critic the parameter size of the current answering model (e.g., this is an 8B model).
• Provide real-time performance: inform the Critic of the current answering model’s average success rate on the training set (e.g., win rate 0.29).
• Ask the Critic to infer the answering model’s capability based on this information and the partially generated answer, point out errors, and finally output an evaluation score.

This mechanism avoids forcing the Critic to memorize all policy information in its weights, greatly improving value estimation accuracy.

Two-stage training: first SFT on data generated by GPT-5, then freeze the Actor and use REINFORCE to train the Critic, with reward Rv(s,z)=1-(r-v̂)². In the RL pretraining phase, GAE’s hyperparameter λ is set to 1, effectively Monte Carlo returns.

Because the critic itself may not be reliable initially, using bootstrapped targets from PPO is inferior to using the actual reward. During joint training, more sophisticated advantage estimation is performed.

During joint training, to avoid writing lengthy reasoning for every token, the system splits the Actor’s full response into logically meaningful segments (e.g., each step in mathematical reasoning).

For each segment, the system concatenates the “original question,” the “Actor’s partial answer written so far,” and the “Actor’s current average win rate (CC information),” applies a default template, and queries the Critic.

After the Critic outputs its thought and predicted score, the system compares the Critic’s “predicted win rate” at this step with the final true “answer result” to compute how accurate the Critic’s scoring was, assigning a reward score to the Critic’s reasoning process.

During advantage estimation, since the Critic scored at the “segment” level, the system replicates that segment’s predicted score to every token within the segment. Combining the final true reward and each token’s score, the system computes each token’s advantage via GAE (Generalized Advantage Estimation).

Experimental Findings

This section aims to answer three questions: Is the generative Critic accurate? Can it truly help the Actor improve? Why is it more stable than traditional critics?

The generative Critic comprehensively outperforms traditional critics. As model parameters increase, its scoring accuracy consistently improves (excellent scaling behavior), and performance is very stable across different random seeds, completely solving the traditional Critic’s problems of “no improvement with larger models” and “extreme fragility.”

In RL, GenAC (Generative Actor-Critic) achieves the best overall performance and requires fewer training steps.

• On mathematical reasoning tasks, starting with Qwen3-8B-Base initial policy, training with DeepScaleR dataset, testing on six math benchmarks.
• Comparisons with GRPO, RLOO, VC-PPO, GenAC: value model methods generally are more sample-efficient than value-free methods. However, if the value model is a traditional discriminative version, it tends to “hit a ceiling” later.
• GenAC improves fastest, is most sample-efficient, and as training steps increase, when all other methods plateau, GenAC’s performance continues to climb, widening the gap.

Analytical experiments

1. Even when taking GPT-5 (the largest model) with a prompt-based value estimation, it is still difficult to become an accurate value function. Second, value modeling is not a typical “LLM-as-a-Judge” task. It strongly depends on the context of “who is the current actor” and “what stage this is.”
2. Generative Critic’s top-1 ranking accuracy significantly surpasses discriminative critics. The gap widens as the candidate set grows. Discriminative critics lose accuracy quickly as candidates increase, even approaching random. The generative critic declines more slowly, maintaining stronger discriminative power.
3. Preliminary evaluation of value models on different distributions: from in-distribution training data to harder, more variable, and even cross-domain data. In-distribution, the generative critic is only slightly better. Once data drifts from training distribution, its advantage becomes dramatic. On AIME24 and GPQA, the directed performance drop is very large, more than half in some cases.

OPID: Intra-Policy Skill Distillation for Agent RL

Existing self-enhancement or fine-grained skill enhancement methods, while providing stronger supervision, often rely on: external skill libraries; searched in-context examples; pre-maintained skill memories. These have two obvious problems:

• High maintenance cost: constantly inserting, updating, deleting, and searching skills.
• Inconsistency with current policy distribution, especially in multi-turn interactions. Slight policy drift can render retrieved skills inapplicable.

OPID first distills “global skills” and “key-step skills” from complete trajectories generated by the current policy itself. Then, based on state importance, it selects the most suitable skill, converts skill discrepancy into token-level additive advantages, and performs PPO updates combined with group-relative outcome rewards.

Extracting Skills from Completed Trajectories

The key innovation is not relying on an external skill library but directly extracting skills from trajectories sampled and completed by the current policy itself. The “current policy” is crucial because these trajectories come from the current model’s actual running distribution, making extracted skills more aligned with the current state distribution, reducing the “skill-state mismatch” problem.

OPID breaks trajectory knowledge into two layers:

1. Trajectory-level skill: Summarizes global patterns of the entire trajectory, such as:

   • How such tasks typically progress.
   • What workflows are usually followed on success.
   • What common errors to avoid on failure.
   • Broad and stable, suitable as default guidance.

2. Step-level skill: Describes local decision knowledge at certain key moments, such as:

   • Avoid repeating ineffective actions.
   • Which object to check next.
   • When to revise sub-goals.
   • When to stop exploring.
   • More fine-grained, but only appears at a few key positions and is highly state-dependent.

For a completed trajectory, the system first organizes it into a segmented record, then uses an LLM-based analyzer to convert the record into natural language skills:

• One trajectory-level skill per trajectory.
• Identify several critical moments and generate step-level skills for those positions.

These marked critical positions form a sparse set, indicating “these places are particularly important.”

• If the current step is a critical moment, use step-level skill.
• Otherwise, fall back to trajectory-level skill.

After routing, OPID injects the selected skill back into the history to form a skill-augmented context. This injection is deterministic, either prepended or appended to the history, as long as original state information is preserved. Key point: not generating a new response, but having the old policy “re-score” the same already-collected response.

Thus, for the same token, there are two probabilities:

• Probability under original history.
• Probability under history with skill added.

The difference gives a signal of “whether this token is supported by the skill.” For each token, OPID defines a skill-additive advantage:

• If adding the skill increases the token’s probability, the token is more consistent with the skill and should be encouraged.
• If probability decreases, it’s less consistent and should be suppressed.

This difference times a valid token mask (only for tokens that actually participate in generation) yields the final signal.

The final loss uses group-relative outcome advantage plus skill-additive advantage. After training, these skills are largely internalized into the model, and inference proceeds as normal.

Experimental Findings

1. Compared to GRPO (which only looks at final outcome reward), OPID shows consistent improvement across most model sizes and tasks. The improvement is especially significant on smaller base models, indicating that extracting skills from trajectories helps weaker models more.
2. The experiment not only compared against GRPO but also against various skill-enhancement and self-enhancement methods like Skill-GRPO, GRPO+OPSD, Skill-SD, RLSD, SDAR. OPID matches or exceeds these strong baselines on most total score metrics.
3. Skill-GRPO and similar methods suffer from significant performance degradation when skill prompts are removed during inference, sometimes even underperforming plain GRPO. OPID maintains a clear advantage even without skill input at inference.
4. In early training, OPID and GRPO both improve; but in later stages, GRPO plateaus while OPID continues to advance. Meanwhile, OPID’s average trajectory length is notably shorter, indicating faster task completion.
5. Particularly useful when data is scarce: OPID’s sample efficiency is clearly higher. The less data, the bigger the advantage over GRPO. With about 60% of data, OPID nearly matches GRPO’s performance with full data. With 80% data, OPID even surpasses GRPO’s full-data results.
6. Hierarchical skills are indeed effective; removing either hurts. However, simply concatenating global and step-level skills without routing performs worse. Using the “prioritize step-level at key points, default to global elsewhere” strategy significantly improves scores. This shows that more skills isn’t always better; the key is using the right granularity at the right place.

Summary

GLM 5.2 addresses the failure of GRPO after long-trajectory compression — when extremely long traces are split, different rollouts produce sub-traces of varying number and length, making intra-group relative ranking unreliable.

It switched to critic-based PPO for token-level advantages, introduced a two-stage anti-cheating (rules + LLM) judge that only intercepts specific violations rather than terminating entire trajectories. The back-and-forth from PPO to GRPO and back to PPO reflects the same insight: the longer the trajectory, the less reliable value-free credit assignment.

Qwen’s verification system covers four task types: SWE uses an agentic quality judge to clean “broken” tasks (many zero-resolution tasks aren’t truly hard but have poor quality), and a trajectory monitor reduced cheating from 28.57% to 0.56%; front-end uses a 25.9-item rubric + interactive judge (simulating real user clicks) to crush code-stacking cheating.

Real-world tasks extract implicit user feedback and train with Span-KTO, revealing that users rarely praise but blame accurately (81.8% high confidence), and discarding negative feedback makes the model dumber (41.8% → 37.2%). Ultra-long-horizon tasks rely on a judge agent that writes its own tests, iterated through five versions to cure common ailments. Key takeaway: verifier and generator must co-evolve; a fixed verification system will eventually be broken by a stronger model.

GenAC reveals that the problem with traditional critics is not just training techniques: the value function itself can be complex (some generation MDP value computations reach P-complete