rlhf

本文提出 Contextual Credit Policy Optimization (CCPO)，一种无需价值critic的步级信用分配方法，通过上下文相似度加权的同伴比较与未来进度信号，为LLM智能体在长程任务中的强化学习提供更精细的优势估计，在ALFWorld和WebShop上显著优于GRPO。

该论文提出 HICO-Bench 基准，包含 180 个日常协调场景，通过配对世界中可验证的结果来评估社交代理，并提出 PaCPO（Paired-Credit Policy Optimization）方法，将 Qwen3-8B 的 Verified Success 从 20.4% 提升至 28.3%。

An OpenAI safety researcher warns that pursuing recursive self-improvement without fundamental alignment progress since RLHF is an unprecedented folly, highlighting the gap between AI capabilities and safety research.

作者提出 LLM 幻觉源于 RLHF 强制模型在 Token 1 无条件承诺（Promise），导致生成过程中无法反悔只能编造 API；建议用 Winograd/Flores 对话行动状态机的『Zero-Shot Flores Handshake』显式授权模型先反问或拒绝，再生成代码。

提出一种无需人工偏好标注的多模态奖励建模方法：在冻结的 Qwen3-VL 骨干网络的中间层上训练线性探针，通过扰动字幕对比进行训练，在多个基准上达到或超过 GPT-4o，且微调计算量仅为千分之一。

微软发布 AesCode-8B 与 AesCode-32B 模型，可依据图像美学参考生成信息丰富的可视化产物（如幻灯片、海报、仪表盘）为结构化 HTML/CSS 代码，通过图结构监督与解耦的跨模态奖励解决视觉布局与内容准确性的权衡。

A research paper exploring KL-regularized policy optimization, likely relevant to RLHF and reinforcement learning from human feedback, posted on arXiv.

LatentGRM 提出了一种基于语义分块与压缩重建的潜在评估框架，通过在连续隐空间中完成 rubric 引导的成对偏好判断，将评估轨迹压缩 8.9–9.2 倍并把推理时间减少 6.1–7.0 倍，同时保持与显式 SFT 评委相当的偏好准确率。

ConvoDrift introduces a multi-turn conversational dataset for modeling progressive stylistic tone drift under fixed semantic intent, comprising 15,727 conversational structures with style drift labels and persona-conditioned preference pairs, validated via human annotation, LLM-as-judge, and lexical/semantic evaluation.

A paper from Meta Superintelligence Labs shows that pre-trained LLMs equipped with a light inference harness can act as capable agents and even outperform RL post-trained counterparts when given a sufficient test-time sampling budget.

MatrixReward proposes constructing rewards for open-ended generation from a rollout-by-rubric win-rate matrix, using column spread and correlation statistics to derive data-dependent rubric weights and ideal profiles. Evaluated with Qwen3-8B on four benchmarks, it achieves 63.02 average score, outperforming the strongest baseline by ~2%.

The ProVer paper targets pivotal decisions for credit assignment in agentic RL: an LLM judge compares successful and failed rollouts to propose a key trajectory segment, then rollouts sampled before and after that segment estimate its advantage. It improves over GRPO by 9.91% and 7.12% on Qwen3.5-2B and Qwen3.5-4B across ALFWorld, WebShop, and SearchQA.

Hugging Face released a comprehensive guide on multi-harness reinforcement learning, drawing attention from the AI community.

EPFL and University of Helsinki researchers introduce Uncertainty-Normalized Margin DPO (UNM-DPO/ULNM-DPO-WR), which combines preference-strength margins with a learned prompt-dependent uncertainty scale and length normalization, achieving strong win rates over DPO and SimPO on HelpSteer2/3 and AlpacaEval with a Llama-3.1-8B-Instruct policy.

The paper revisits scaling laws for reward optimization in AI alignment, showing that performance scales as Θ(√min{log(M), K}) where M is the amount of preference data and K is the KL-divergence budget, and provides an information-theoretic model plus extensive empirical validation with proxy reward models.

该论文提出 Robust Nash Alignment，一个面向不确定成对偏好的博弈论对齐框架，通过主-对偶代理博弈与乐观镜面下降-上升算法，实现了对最坏情况胜率的认证下界，并在 LLM 对齐实验中优于基线方法。

MetaRubric 通过交替进行证据感知的策略优化与响应引导的评分标准适配，解决基于评分标准的强化学习中「空洞奖励（Vacuous Credit）」问题，在 PubMedQA 上比静态评委 GRPO 提升 6.00–20.40 个百分点。

The paper proposes position-selective self-distillation for training LLM judges from natural language feedback, using per-position entropy shifts to mask memorization-prone tokens and improve out-of-distribution generalization over outcome-supervised RL methods like GRPO by 2–9 points on subjective tasks.

This paper formalizes how collapsing LLM-judge rubric rewards to binary pass/fail hides criterion-level quality changes from the policy's reward signal, showing under a joint-Gaussian model that at least three grading levels are needed to remove this ambiguity, with experiments on MATH and SciBench confirming that binary stretches are invisible but detectable with three grades.

The paper introduces Density-Aware Reward Aggregation (DARA), which corrects reward-wise normalized multi-reward RL with an inverse-square-root density weighting based on active-group density, reducing signal imbalance between rewards. Experiments on tool calling and math reasoning show DARA reaches high format/length compliance in up to 26% and 65% fewer training steps than GDPO.