@xennygrimmato_: if you’re wondering how token-level rejection sampling works in this paper, here’s how they do it: M_t = max_v [ pi_the…
Summary
Explains token-level rejection sampling for RLHF/PPO, where importance ratio M_t is the maximum over vocabulary and tokens are accepted with Bernoulli sampling based on w_t / M_t.
View Cached Full Text
Cached at: 07/11/26, 07:28 PM
if you’re wondering how token-level rejection sampling works in this paper, here’s how they do it:
M_t = max_v [ pi_theta(v) / pi_old(v) ] z_t ~ Bernoulli( w_t / M_t ) w̃_t = 1 if z_t=1 else w_t L(θ) = E[ Σₜ w̃_t(θ) · Âₜ ]
M_t is the max importance ratio across the vocabulary. w_t is the importance ratio for the i-th token. Âₜ comes from GRPO/PPO.
Rosinality (@rosinality): https://t.co/o4bRt58hDD
Using rejection sampling to consider accepted tokens as on-policy ones.
Similar Articles
Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling
Bebop proposes entropy-aware multi-token prediction with rejection sampling and a novel TV loss to accelerate RL training of LLMs, achieving up to 1.8x speedup. The method addresses the degradation of acceptance rates during RL by optimizing training objectives.
Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
This paper proposes a reinforcement learning framework that improves LLM reasoning efficiency by modeling token significance to selectively penalize unimportant tokens while preserving essential reasoning, using both significance-aware and dynamic length rewards to reduce verbosity without sacrificing accuracy.
S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF
S2T-RLHF proposes a sentence-to-token reward decomposition framework that improves training stability and robustness in preference-based RLHF by assigning sequence-level rewards at sentence granularity, avoiding the instability of overly fine-grained token-level refinement.
Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning
This paper introduces CPPO, a method that improves reinforcement learning with verifiable rewards for LLMs by using position-weighted thresholds and cumulative prefix budgeting to address limitations of uniform token-level trust regions.
Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning
Introduces Independent Combinatorial Tokens (ICT) framework that uses Jensen-Shannon divergence between token logit distributions to identify critical branching points, preventing entropy collapse and explosion in RLVR for LLM reasoning. Achieves up to 14.9% pass@4 improvement on Qwen models.