S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning
Summary
S3AC 提出一种稳定的 Actor-Critic 训练方法与难度感知提示采样（DAPS），在不依赖分组 rollout 的情况下改进大模型的长链推理，性能优于 GRPO-X 等现代基线，并可在 35B 模型上维持单 rollout 训练。
View Cached Full Text
Cached at: 10/28/26, 01:59 AM
Paper page - S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning
Source: https://huggingface.co/papers/2610.35518
Abstract
S3AC introduces a stabilized actor-critic training method with difficulty-aware prompt sampling to improve long reasoning in large language models without relying on grouped rollouts.
Critic-free methods are widely used inreinforcement learning with verifiable rewards(RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lacktoken-level credit assignmentand typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initializedvalue head. We introduce S3AC (Stabilized, Selective, and ScalableActor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we proposeDifficulty-Aware Prompt Sampling(DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense andmixture-of-expertsmodels from 1.5B to 397B parameters, outperforming a modernGRPOvariant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.
View arXiv pageView PDFGitHub5.22kAdd to collection
Get this paper in your agent:
hf papers read 2610\.35518
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.35518 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2610.35518 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.35518 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Simple Actors and Deep Critics for Scalable Reinforcement Learning
This paper proposes LAC (LightActor, deepCritic), an offline reinforcement learning method that allocates model capacity to the critic for improved inference efficiency, matching performance of complex generative actors with lower latency.
@tengyuma: Always thought GRPO-style RL can't be optimal. With billions going into self-play & RSI, the underlying RL methods need…
Tengyu Ma highlights AC2 (Actor-Critic with Action Chunking), a reinforcement learning method that uses a learned critic to score token chunks, enabling partial rollouts and faster training than GRPO for LLMs.
ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
This paper introduces ACSAC, a reinforcement learning method that uses an adaptive chunk size actor-critic algorithm with a causal Transformer Q-network to handle long-horizon, sparse-reward tasks. It demonstrates state-of-the-art performance on manipulation tasks by dynamically adjusting action chunk sizes based on state-dependent needs.
Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping
The paper proposes Li2SAO, a lightweight asynchronous RL method for LLM post-training that uses group-relative actor-critic bootstrapping to warm up the critic before switching to single-rollout GAE, catching up with or surpassing GRPO early in training and raising later performance ceilings.
Z.ai's Stable Asynchronous RL (13 minute read)
The paper introduces Single-rollout Asynchronous Optimization (SAO) to address stability and off-policy challenges in asynchronous RL for LLM post-training, and demonstrates that SAO consistently outperforms GRPO on agentic coding and reasoning benchmarks.