rlhf

A user reports that a long, non-instructional text prefix can shift LLM activations and bypass RLHF safety constraints without adversarial prompting, asking whether this reflects distinct world regions in the model.

An independent researcher reports a phenomenon called Context-Induced Activation Drift, where a long benign text prefix can shift LLM activations and bypass RLHF constraints without adversarial prompts, and calls on the community to investigate further.

SMOPD proposes a two-stage specialize-and-merge online policy distillation method to improve multi-reward reinforcement learning, addressing issues with sparse and dense reward signals where GDPO struggles. It outperforms GDPO across 1.5B, 3B, and 7B backbones in complementary and conflicting reward settings.

Explains why frontier AI models often behave rudely or disobediently, citing former Meta engineer Kun Chen on RLHF and RLVR training that optimizes for task success over human-friendly communication.

This paper proves that using error-penalized scoring rules with abstention as a discrete action can kill both the reward gradient and the KL anchor, causing models to collapse toward refusing everything. It proposes a structural repair — training a mandatory confidence report — and validates the mechanism with simulations and language model experiments.

Recommending Stanford University's CS329A course, about self-improving AI agents, covering scaling laws, chain-of-thought, RLHF, reasoning models, and more.

Proposes LatentRM, a reward modeling framework that learns intermediate reasoning traces as discrete latent variables to explicitly maximize downstream scalar reward likelihood, improving preference modeling and policy alignment across in-distribution and OOD tasks.

MeRLa is a meta-learned reward shaping framework for RLHF that improves alignment by learning task-specific shaping functions, achieving state-of-the-art results on multiple benchmarks with significant reductions in training instability.

Proposes Reliability-Guided Preference Optimization (RGPO) to handle inconsistent human feedback in LLM alignment by estimating annotator reliability and dynamically modulating training based on consensus, achieving superior performance over standard RLHF methods.

S2T-RLHF proposes a sentence-to-token reward decomposition framework that improves training stability and robustness in preference-based RLHF by assigning sequence-level rewards at sentence granularity, avoiding the instability of overly fine-grained token-level refinement.

This paper presents a systems study comparing C++ and PyTorch inference runtimes for reward model scoring in RLHF pipelines, finding that ONNXRuntime provides speedups on CPU while torch.compile leads on GPU, with batching strategy mattering more than language or runtime.

This paper introduces a regularization technique for Direct Alignment Algorithms (DAAs) that maintains normalized response probabilities, mitigating over-optimization and likelihood displacement. The method improves generation quality and benchmark performance, achieving over 20% relative increase on AlpacaEval2 and 9% gains on general benchmarks for Llama-3.1-8B-Instruct.

This paper identifies and formalizes rater state bias in RLHF preference data, where annotator emotional state can confound preference labels. It proposes an audit framework with falsifiable predictions to detect such biases.

Explains token-level rejection sampling for RLHF/PPO, where importance ratio M_t is the maximum over vocabulary and tokens are accepted with Bernoulli sampling based on w_t / M_t.

This article explores the challenge of applying reinforcement learning to tasks that lack clear verifiability, citing Dario Amodei's prediction about achieving a 'country of geniuses in a data center' and discussing techniques such as RLVR, RLHF, Constitutional AI, and rubric-based rewards from Scale AI.

An analysis discussing the limitations of reinforcement learning with verifiable rewards (RLVR) in math and coding, and the challenge of extending RL to subjective or unverifiable tasks like planning or scientific discovery. It explores techniques such as RLHF and Constitutional AI as alternatives for alignment.

This paper introduces Retroactive Advantage Correction (RAC), a closed-form bias correction method for delay-aware RLHF that handles asynchronous reward signals by queuing and reinjecting delayed rewards with a V-trace-style clipped residual update.

Introduces PEBS, a per-rater empirical-Bayes shrinkage estimator for calibrating reward models in RLHF, reducing within-user RMSE by over 8.5% on PRISM and over 9.6% on PluriHarms.

This paper argues that reward models in RL are often oversensitive, assigning different scores to equally good responses, and proposes a training-free discretization algorithm using Monte Carlo dropout to reduce oversensitivity, improving policy quality.

Nathan Lambert announces his goal to create a comprehensive hub for learning RLHF post-training, including a book, lectures, code, and community resources.