@ziv_ravid: Just read the new paper from Tsinghua/Z.AI on async RL for agents (arXiv:2607.07508). It comes several weeks after the …

X AI KOLs Timeline Papers
async-rl reinforcement-learning agents critic glm-5.2 tsinghua

Summary

Discusses a new paper from Tsinghua/Zhipu AI on asynchronous reinforcement learning for agents, and notes that their previous GLM-5.2 model uses a critic instead of GRPO.

Just read the new paper from Tsinghua/Z.AI on async RL for agents (arXiv:2607.07508). It comes several weeks after the release of GLM-5.2, in which they mentioned that they use a critic instead of GRPO. 🧵 https://t.co/upDfcrEMao
Original Article
View Cached Full Text

Cached at: 07/11/26, 09:22 AM

Just read the new paper from Tsinghua/Z.AI on async RL for agents (arXiv:2607.07508). It comes several weeks after the release of GLM-5.2, in which they mentioned that they use a critic instead of GRPO.

Their claim is that GRPO doesn’t fit asynchronous training. A group has to wait for its slowest rollout, and with 128k-token agentic trajectories, that wait is long, so your data goes stale before you train on it.

Their answer is one rollout per prompt. Which means you need a value model again (a critic!). Fun to see, because critics were the classical way to do RL (PPO), but they increase the memory and are hard to train well, so people ditched them for GRPO and its group-mean baseline.

To make the critic trainable, they freeze the attention layers and only update the MoE projections (the gradient blowups come from the attention layers). They update the critic twice per policy step. And for multi-turn trajectories, they skip observation tokens in GAE.

Agent trajectories interleave model actions with environment observations. Standard GAE propagates advantages through all of it, including tokens the model never produced. So they connect the last token of one action directly to the first token of the next

For the off-policy problem, they just drop the old policy entirely. The importance ratio is current policy over the rollout engine’s logprobs, and anything outside a hard two-sided band gets masked out of the gradient without being clipped.

They show that GRPO collapses faster, while SAO not and beats GRPO variants on SWE-Bench Verified and the math benchmarks. There’s also a nice experiment where the reward preference shifts mid-training and the value model tracks it much faster than a running-mean baseline.

Anyway, very impressive work showing the mismatch between group sampling and asynchrony is real, and I expect more papers on this.

Similar Articles

Z.ai's Stable Asynchronous RL (13 minute read)

TLDR AI

The paper introduces Single-rollout Asynchronous Optimization (SAO) to address stability and off-policy challenges in asynchronous RL for LLM post-training, and demonstrates that SAO consistently outperforms GRPO on agentic coding and reasoning benchmarks.

@VukRosic99: How Is GLM 5.2 Trained? Tsinghua's Async RL Paper Explained The paper from Tsinghua University replaces GRPO's wait-for…

X AI KOLs Timeline

This paper from Tsinghua University introduces Single-rollout Asynchronous Optimization (SAO) for reinforcement learning post-training of LLMs. SAO replaces batch-based GRPO with single-rollout asynchronous training to reduce idle GPU time and improve stability, and it was used to train the GLM-5.2 model (750B-A40B), achieving state-of-the-art results on agentic coding and reasoning benchmarks.