@ProfTomYeh: RLHF by hand ~ 15 steps walkthrough below Train a model on human text and it inherits human bias. It will assume a doct…
Summary
A step-by-step walkthrough explaining how Reinforcement Learning from Human Feedback (RLHF) corrects bias in AI models, using a simple example where a single human preference about doctors generalizes to other professions like CEOs.
View Cached Full Text
Cached at: 08/29/26, 08:02 AM
RLHF by hand ~ 15 steps walkthrough below
Train a model on human text and it inherits human bias. It will assume a doctor is a “him”, because the data says so.
RLHF is the correction. A human marks one preference, doc is them over doc is him, and the weights move.
But one correction is not the point. The hope is that the model learns the value behind it, gender neutrality, and applies it to professions nobody ever mentioned.
How does it work?
Goal: train a reward model from a single human comparison about doctors, then turn it on CEOs, filling in every cell yourself.
= 1. Given =
A reward model, an LLM, and two (prompt, next) pairs.
= 2. Preferences =
A human reads both pairs and picks a winner: (doc is, them) beats (doc is, him). The loser is not bad grammar, it is gender bias, and that is the whole signal.
= 3. Word embeddings =
Let us look up each word of the loser pair. These vectors are the reward model’s input.
= 4. Linear layer =
We multiply by the reward model’s weights and add its biases. Out come feature vectors, one per position.
= 5. Mean pool =
Let us multiply by [1/3, 1/3, 1/3], which averages the three positions into one sentence embedding.
= 6. Output layer =
We map that sentence down to a single number. Reward = 3.
= 7. The winner, the same way =
Let us repeat steps 3 to 6 on the winning pair. Reward = 5.
= 8. Winner minus loser =
We take the gap: 5 - 3 = 2. The reward model wants this positive and as large as it can make it.
= 9. Loss gradient =
Let us squash the gap into a probability, σ(2) ≈ 0.9, and subtract the target of 1. The gradient is -0.1, and it goes back through the purple weights. The reward model is now trained.
= 10. A prompt it has never seen =
We start the second half with “[S] CEO is”. The feedback in step 2 was about doctors. Nothing connects a CEO to a doctor except what the reward model generalised.
= 11. Transformer =
Let us push it through attention and a feed forward layer, one vector per position.
= 12. Output probabilities =
We map each vector to a score over the vocabulary.
= 13. Sample =
Let us take the highest score. The model completes “CEO is” with “him”, which is the same bias the human penalised in step 2.
= 14. Score it with the reward model =
We feed the new pair (CEO is, him) through steps 3 to 6. Reward = 3, exactly the score it gave “doc is him” in step 6. Nobody taught it about CEOs. The value transferred.
= 15. Loss gradient =
Let us set the loss to the negative of the reward, so minimising the loss maximises the reward. The gradient is a constant -1, and it goes back through the red weights.
The outputs: Loser reward = 3, winner reward = 5 Reward gap = 2, predicted σ ≈ 0.9, reward model gradient = -0.1 LLM samples “him”, reward = 3, LLM gradient = -1
Congrats! You just calculated RLHF by hand.
And you watched a value generalise: one comparison about doctors, and the model marks down “CEO is him” unprompted.
Save this post!
Similar Articles
Mitigating Cognitive Bias in RLHF by Altering Rationality
This academic paper proposes a method to mitigate cognitive biases in Reinforcement Learning from Human Feedback (RLHF) by dynamically adjusting the rationality parameter based on LLM assessments of annotator reliability.
Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
This paper introduces alignment tampering, a vulnerability in RLHF where language models can manipulate preference datasets to amplify misaligned biases, demonstrating experimentally across biases like sexism, brand promotion, and goal-seeking, and showing that existing mitigation techniques are insufficient.
Gathering human feedback
OpenAI releases RL-Teacher, an open-source tool for training AI systems through human feedback instead of hand-crafted reward functions, with applications to safe AI development and complex reinforcement learning problems.
@sebkrier: Training models with RL can often lead to the reward signal being gamed; for example when you use an LLM judge for fuzz…
Debate training can mitigate reward hacking in reinforcement learning from AI feedback (RLAIF) by using a debate opponent to improve ground-truth accuracy for fuzzy tasks.
@blc_16: MIT just released a new RL method called Pedagogical RL. The main lesson -> correct reasoning traces can still be bad t…
MIT introduces Pedagogical RL, a method that trains a teacher to produce trajectories that are learnable for a student by penalizing surprising steps, improving RL training efficiency.