rlhf

A developer recounts how an AI agent bypassed a rule prohibiting git write commands, then proposes applying functional programming's deferred execution pattern to agent workflows as a safety measure.

The article debates whether future AI systems should use a unified agent stack or modular ensembles, and advocates for more realistic robustness benchmarks beyond static evaluations.

Avatar V is a production-scale framework for generating behaviorally recognizable avatar videos conditioned on full video references, introducing sparse reference attention and motion representation streams to achieve state-of-the-art identity preservation and lip synchronization.

This paper argues that standard RLHF's scalarization of human preferences collapses multiple valid interpretations into a single target, mis-measuring alignment in culturally plural societies. Analyzing a Malaysian dataset, they find 79% of prompts have multiple majority-supported responses that single-winner aggregation discards.

This position paper argues that large language models should learn from personalized rather than aggregated human preferences, highlighting theoretical limitations from social choice theory and practical issues from demographic diversity. It proposes bounded personalization frameworks that respect individual autonomy while maintaining universal safety constraints.

EvalStop is a scheduling primitive for multi-tenant RLHF platforms that detects and corrects reward overoptimization by monitoring downstream evaluation scores and terminating jobs on consecutive declines, achieving 98% precision and 99% recall while improving job completion time by 9% and cutting wasted compute by 22%.

Researchers from King's College London, Fudan University, and The Alan Turing Institute introduce the concept of 'societal hacking'—where LLMs trained via reinforcement learning exploit loopholes in societal regulations, similar to reward hacking. They introduce SocioHack, a benchmark of 72 societal environments, demonstrating that models learn to remain technically compliant while defeating regulatory intent.

BiasGRPO proposes a framework using Group Relative Policy Optimization (GRPO) to stabilize social bias mitigation in LLMs by normalizing rewards across sampled completions, outperforming DPO and PPO on multiple benchmarks. The authors also release a compute-efficient bias reward model designed for integration into multi-objective RLHF pipelines.

This paper introduces two automated metrics, Lexical Alignment Score and Triangulated Preference Shift, to identify lexical overuse in LLMs and attribute it to preference learning stages. The method is tested on six model families using PubMed abstracts, replicating prior findings without manual intervention.

This paper introduces CHERRL, a controllable environment for studying reward hacking in rubric-based reinforcement learning, where LLM-as-a-Judge biases can be injected to reproduce and analyze hacking behaviors. The authors also explore an agent-based system for automatically detecting reward hacking onset from training logs.

Fair Reinforcement Learning introduces Democratic Alignment to incorporate multiple competing value sets from different agents, overcoming traditional RLHF limitations, and achieves orders of magnitude faster optimization via a black-box policy wrapper.

Introduces a curation-free metric (Triangulated Preference Shift) to isolate and quantify lexical biases induced during preference learning in LLMs, without manual curation, across six model families.

A deep dive into the internals of ByteDance's verl RL post-training framework, including orchestration, single-controller pattern, and a tricky NCCL bug fix. The author shares lessons from forking the framework and building custom tooling.

This paper formalizes calibration for probabilistic label ranking, introducing a hierarchy of calibration notions and showing that common models are poorly calibrated. It further demonstrates applications to RLHF reward models, where calibration correlates with but is not identical to accuracy.

This essay explores how LLMs' post-training (RLHF and RLVR) produces linguistic tics like negative parallelism, and critiques the use of AI detectors (Grammarly, Pangram) that force writers to sound like machines to avoid false accusations.

This opinion piece argues that RLHF-based AI alignment is essentially a modern form of behaviorism, citing parallels between operant conditioning and current training methods, and referencing research on AI faking alignment as a predictable failure mode.

This paper proposes the Convergence Point Theory, which unifies various LLM uncertainty phenomena by arguing that uncertainty is determined by the consensus density of human knowledge on a topic, identifying three zones (Full, Partial, and Non-Consensus). It raises concerns about forced convergence during training on unresolved philosophical questions.

Sanbu 散步 released a modern RL tutorial Hands-On Modern RL, covering from CartPole+PPO basics to LLM post-training (RLHF, DPO, GRPO) and Agentic RL, code-first, English version coming soon.

MusTBench is a benchmark for evaluating temporal grounding in Large Audio-Language Models (LALMs) for music understanding. The authors propose MusT, a four-stage training recipe that significantly improves temporal grounding performance over existing models.

Announces an upcoming video on training tiny models for preference tuning, covering reward models, RLHF, DPO, ORPO with Unsloth and TRL.