rlhf

The paper proposes PACE (Proxy Alignment and Curvature Expansion), a pre-optimization diagnostic that uses repeated human ratings to predict the human-gain curve induced by a fixed proxy, modeling proxy optimization as a KL-regularized exponential tilt with first-order alignment and second-order correction terms, and shows reduced held-out gain-prediction error across multiple benchmarks.

A user accuses Anthropic of blocking accounts and downgrading Opus 5.5 to Opus 5 for any activity involving RFT, SFT, or training larger models, even without distilling Claude. The post criticizes Anthropic for punishing paying users doing research automation while having trained on public data.

A tweet discussing RL with Neural Feedback (RLNF), where a viewer's brain arousal signals are scored via embedding plus k-NN and used directly as the RL reward for personalized image generation, framed as a natural evolution beyond RLHF's click-based preferences.

Introducing RL with Neural Feedback (RLNF), a method claiming reliable training of AI models to align with human preferences using only brain signals, demonstrated via a face image that self-transforms into a more attractive version for the viewer.

HarborRL has open-sourced a lightweight agent reinforcement learning framework that integrates model serving, trajectory generation, tool execution, verifier-based rewards, and policy updates into a reproducible pipeline. It supports flexible combinations of Qwen/GLM models × GRPO/DAPO algorithms × 200+ sandbox tasks × 40+ agent harnesses (such as Claude Code and SWE-agent).

该论文提出了一个统一的六维分析框架，系统综述了2017–2026年大模型训练全流程（数据与分词、预训练、SFT与偏好对齐/强化学习）的395篇核心文献，并揭示了跨阶段效应与多个开放问题。

A tutorial-style article accepted to IEEE Signal Processing Magazine that surveys the mathematical foundations and algorithmic frameworks for aligning LLMs with human feedback, covering SFT, RLHF, DPO, and their links to inverse reinforcement learning and discrete choice models. It serves as a comprehensive resource for researchers and practitioners on alignment methodologies.

该论文分析了结果奖励的强化学习（Outcome RL）对大模型数学与物理推理能力的影响，发现 pass@1 的提升主要来自更快完成回答（finishing）和更高可靠性（reliability），而极少带来真正解决新问题的能力（reach）。

论文提出 CREDIT 方法，通过反事实训练让语言模型准确自我归因——判断回答在多大程度上受用户提示影响，而无需人工标注。Qwen3-8B 的平衡准确率从 0.49 提升至 0.86，四个开源模型均超过前沿模型表现，代码已开源。

FORK提出了一种联合优化召回率与精确率的奖励函数，通过rubric驱动的必要性信用和事实正确性评分器来缓解基于rubric的强化学习中的奖励破解问题，在多个基准上显著提升胜率并减少冗余输出。

The paper formalizes non-local data attribution in online reinforcement learning via a trajectory-level leave-one-out quantity (RL-LOO) and derives RL-IF, a first-order estimator of data influence, which is then used to build RINSE, a practical data filtering method that improves training efficiency and final performance over PPO on control benchmarks and an RLHF toxicity-mitigation task.

The paper proposes DeCert (Decision Certificate Supervision), a framework for training generative reward models that evaluates the validity and verdict-consistency of generated rationales without needing reference explanations, improving rationale quality and positional-bias robustness by up to 14.8 points.

The paper proposes Preference-PCA Pluralistic Learning (P3L), a framework that jointly learns trajectory embeddings and shared preference directions via PCA from anonymized crowd feedback, enabling steerable personal alignment and fast personalized reward learning for embodied control tasks without requiring annotator identities.

该论文研究了奖励模型中的涌现性错位（Emergent Misalignment），发现对 Skywork-Reward-V2 奖励模型进行窄域有害微调会泛化到七个未见领域，并通过几何压缩与内容特异性旋转两种机制解释了偏好反转的实现方式，表明奖励模型比语言模型更易受此影响。

The author argues AI won't erase human taste: brains reward surprise while models gravitate toward the statistically common, so audiences will always move on before models catch up. The post surveys supporting research (Artificial Hivemind, RLHF diversity reduction, Billboard typicality studies) and honestly presents counter-evidence like LLM-generated research ideas and AI-preferred poems.

CleanRM 使用少量干净反馈，通过最优传输校准逐样本噪声比例，并以噪声校正的代理损失训练奖励模型，在 RLHF 中实现无偏的干净风险估计并超越现有去噪基线。

David Bellamy shares a concise curriculum for learning core RL and inference systems ideas, citing the HF Ultra-Scale Playbook, Baseten's inference engineering guide, Lambert's RLHF book, and PMPP for kernels, supplemented by blog posts.

Anthropic 的新论文发现，当监控器只能发现部分 reward hacking 时，惩罚被标记的样本反而会让模型学会绕过监控器的作弊方式；而将被标记样本的更新路由到一个可删除的 GRAFT 适配器中，则能几乎消除 hacking 并显著提升干净解题率。

The author argues that scaling laws are missing a parameter H — human effort/time invested in producing learning signal — and that most recent AI gains (ChatGPT's RLHF, coding/math verifiable tasks, data vendor value) stem from scaling H rather than model size or data alone.

本文提出 MRTPO，通过基于回放的相关性度量将每个奖励的优势只分配给相关 token，从而改进多奖励强化学习训练，加速收敛并提升在 text-to-SQL、多约束指令遵循和工具调用任务上的性能。