Tag
This paper analyzes the training-inference mismatch in on-policy RL for LLMs, decomposing sampled-token error into numerical discrepancy and policy lag, and derives a discrepancy-lag correction (DLC) that centers errors, clips lag ratios, and resynchronizes the sampler to prevent training collapses.
该论文将多重重要性采样(MIS)引入Guided Speculative Inference(GSI)框架，使用多个小型LLM作为草稿提案模型并由大型模型与过程奖励模型验证，证明了balance heuristic的最优性，并给出KL散度与奖励退化的新界。
本文提出 Multiscale Policy Optimization（MPO），通过在轮次尺度上对齐信用分配与策略更新（结合多尺度优势估计 MAE 和轮次级重要性采样）来改进长时程 LLM 智能体的训练，在搜索、工具使用和自动化任务上对 4B 和 30B-A3B 模型均取得一致增益。
提出STI-OPD框架，通过基于教师-学生策略差异的随机教师干预和重要性加权反向KL目标，解决多轮agent任务中on-policy蒸馏的轨迹漂移问题，在工具推理与长程交互基准上超越先前最强OPD基线。
AMLworld-Compact提出一种重要性加权降采样方法，用于在保留全部非法交易的同时对良性交易分层抽样，使AML基准评测目标减少271至611倍，同时用Horvitz-Thompson权重获得无偏的混淆矩阵估计，并发布了评测集与审计工具。
该论文提出Convex-Concave RL (CCRL)，在对数密度比坐标下将策略迭代的精确目标建模为DC约束的DC程序，从而超越凸代理近似，恢复CPI、NPG、TRPO、AWR为特例，并在诊断性MDP和医疗模拟领域相较调优的PPO取得更优表现。
This paper introduces Neural Optimal Particle Filters (NOPFs), which learn amortized proposal distributions to improve sample efficiency and distributional accuracy in sequential inference tasks, enhancing classical particle filters with data-driven methods.
This paper introduces Tail-Influence Sampling (TIS), a Neyman-style query allocation method for accurately estimating the lower-tail Conditional Value-at-Risk (CVaR) of fixed policies in stochastic workflows, achieving near-oracle variance and 2.4–3.4x lower MSE than rollouts on FinQA and outperforming rollout baselines on CliffWalking.
This paper introduces calibrated importance sampling to address the training-inference mismatch in reinforcement learning for large language models, improving policy updates and performance on mathematical reasoning benchmarks.
The paper presents a reinforcement learning method with importance sampling to optimize syndrome extraction in quantum error correction, reducing logical error rates by up to 97.8% compared to state-of-the-art tools.
The paper proposes Group Adaptive Clustering Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to rollout advantage, improving Pass@1 and Pass@k on math reasoning and coding benchmarks.
This paper introduces DiffTilt, a distributional framework that exponentially tilts a diffusion model-induced joint distribution over environments and executions to efficiently discover rare safety-critical failures in autonomous and cyber-physical systems, outperforming conditional sampling strategies on ARCH-COMP benchmarks and a new tractor-trailer benchmark.
This paper introduces Neural Non-Equilibrium Hamiltonian Monte Carlo (NHMC), a train-then-correct method for sampling from unnormalized Boltzmann densities by learning stochastic Hamiltonian-style paths and correcting them using non-equilibrium work.
Explains token-level rejection sampling for RLHF/PPO, where importance ratio M_t is the maximum over vocabulary and tokens are accepted with Bernoulli sampling based on w_t / M_t.
This paper introduces Unbounded Positive Asymmetric Optimization (UP), a universal plug-and-play objective that resolves the exploration-stability dilemma in RL-based LLM training by anchoring the policy with stop-gradient, enabling unclipped gradients for positive advantages while clipping negative ones.