Tag
该论文将多重重要性采样(MIS)引入Guided Speculative Inference(GSI)框架，使用多个小型LLM作为草稿提案模型并由大型模型与过程奖励模型验证，证明了balance heuristic的最优性，并给出KL散度与奖励退化的新界。
This paper identifies 'on-policy audit collapse,' a failure mode where refreshing a component verifier against full-outcome labels on improved-policy data causes the audit to systematically prefer worse verifiers. The authors derive an exact inversion threshold and propose repairing audits by preserving own-failure support in the refresh distribution.
该论文研究了奖励模型中的涌现性错位（Emergent Misalignment），发现对 Skywork-Reward-V2 奖励模型进行窄域有害微调会泛化到七个未见领域，并通过几何压缩与内容特异性旋转两种机制解释了偏好反转的实现方式，表明奖励模型比语言模型更易受此影响。
This paper explains why majority-vote self-rewards match verifiable-reward RL by showing that, at the batch level, incoherent label errors cancel out so the update direction aligns with the gold gradient; it finds error coherence—not error rate—determines when self-generated labels can replace verification.
该论文将 Best-of-N 采样建模为正确与错误响应极端奖励尾部之间的竞争，利用极值理论刻画了三种渐近机制并推导了有限样本缩放定律，解释了奖励 hacking 导致的性能退化，并提出轻量级 tail-tension 准则来判断额外采样何时有害。
Nebius AI 提出 Caddie 方法，通过强化学习从任务成败结果训练 LLM agent 的自然语言批评器（critic），无需逐步标注。Qwen3-4B critic 在 MuSiQue 上将成功率提升逾 25 个百分点，并可跨基座模型与任务域泛化。
The paper revisits scaling laws for reward optimization in AI alignment, showing that performance scales as Θ(√min{log(M), K}) where M is the amount of preference data and K is the KL-divergence budget, and provides an information-theoretic model plus extensive empirical validation with proxy reward models.
The paper proposes position-selective self-distillation for training LLM judges from natural language feedback, using per-position entropy shifts to mask memorization-prone tokens and improve out-of-distribution generalization over outcome-supervised RL methods like GRPO by 2–9 points on subjective tasks.
The paper introduces ContextProgress-Bench, a benchmark showing that embodied Progress Reward Models fail at progress estimation in long-horizon tasks without proper context, and proposes ProgressCompass, an agentic loop that supplies the needed context via general-purpose VLMs, cutting progress error by 63-82%.
BiasReducer is a lightweight framework that edits only the linear reward head of reward models to adaptively mitigate biases toward superficial attributes like response length and confidence, using a sparse-autoencoder-style encoder to detect and rank relevant biases per dataset. Across five reward models it improves three benchmarks by 8.3, 18.0, and 6.9 percentage points on average, outperforming training-based baselines and reducing downstream verbosity and sycophancy.
This paper introduces RM-EVAL, a reward model trained on human preference data for reference-free meta-evaluation of grammatical error correction, and shows how it can improve GEC systems via reward-guided text generation.
The paper introduces a multi-stage architecture for learning individuated utility functions from multi-modal data, showing that models accounting for heterogeneous preferences outperform universal utility models in subjective tasks like aesthetic judgments.
This paper proposes Generative Verification (GenV), a method using generative reward models to detect reference-equivalence failures in autoformalization, addressing vulnerabilities in neurosymbolic systems and improving verification accuracy.
A tweet highlights how someone simplifies complex AI agent concepts like reward models and evaluation harnesses using clear analogies, making them easy to understand without jargon.
The paper proposes TUP, a method for BoN-style distillation via rank-based classification that truncates low-ranked completions and upweights high-ranked ones to improve alignment efficiency and performance.
The paper presents Contribution-Contrast (CoCo), a novel response-level interpretation method for Mixture-of-Experts reward models, which captures routing and preference behavior more faithfully than routing-weight-based approaches.
Proposes LatentRM, a reward modeling framework that learns intermediate reasoning traces as discrete latent variables to explicitly maximize downstream scalar reward likelihood, improving preference modeling and policy alignment across in-distribution and OOD tasks.
OSReward introduces a standardized benchmark for evaluating VLM judges on computer-use agent trajectories, revealing that even state-of-the-art models have systematic leniency bias. The authors release OS-Shepherd-9B and 35B reward models that provide reliable, low-cost rewards for CUA tasks.
CSPF proposes a constrained shared-private fusion method to integrate representations from multiple reward models for non-verifiable preference evaluation, outperforming baselines.
This paper argues that reward models in RL are often oversensitive, assigning different scores to equally good responses, and proposes a training-free discretization algorithm using Monte Carlo dropout to reduce oversensitivity, improving policy quality.