RLHF
Reinforcement Learning from Human Feedback.
For a small example, one answer is concise and correct; another is polished but wrong. Write a rubric, compare independently, and notice how vague preference labels can reward style over truth.
RLHF
In practice
A base model must follow domain instructions safely. Collect diverse comparisons, train or optimize preferences, hold out evaluators, and test capability regressions and reward hacking.
Initializing Interactive Playground...
Experiment notes
Record your experiment
Predict how one control changes the RLHF result.
Implementation
Repair a failing RLHF starter
The starter deliberately follows a tempting but flawed assumption: Treating the reward model as ground truth
- Write a prediction that makes the flawed assumption visible and names what should change.
- Manipulate one playground control while holding the others fixed.
- Save the result and explain why this correction works: Use it as a fallible proxy and retain independent evaluation.
Check: Your saved hypothesis predicts a direction, the experiment tests it, and the result changes or supports your explanation.
Knowledge check
10 questions
Explanations after each answer · 80% to pass · RLHF
Chapter 1 of 9 · Core
The Alignment Problem & SFT Limitations
Large language models trained via self-supervised next-token prediction are highly capable text generators but lack any awareness of human values, truthfulness, or safety. They simply mimic the statistical distributions of their pre-training corpora—including toxic language, structural biases, and factual hallucinations.
The Alignment Problem asks: how can we guide a neural policy to be helpful, honest, and harmless?
Supervised Fine-Tuning (SFT) and Its Limits:
A naive approach is to perform Supervised Fine-Tuning (SFT) using high-quality, human-curated demonstration pairs: While SFT adapts the model's tone and formatting, it suffers from three fundamental limitations:
- Lack of Negative Feedback: Cross-entropy only teaches the model what *to* generate. It cannot penalize toxic or hallucinated behaviors.
- Exposure Bias: Because the model is trained with teacher forcing, error compound cascades quickly at inference when generating long text sequences.
- High Annotation Cost: Writing high-quality, expert level answers is extremely slow and expensive.
The Preference Scaling Insight: According to cognitive psychology, it is dramatically easier and faster for a human annotator to rank and compare two candidate model responses than to write the ideal response from scratch. RLHF exploits this comparative capability.
Key equations
Video explanation
What Is Reinforcement Learning from Human Feedback?
IBM Technology
Follow the role of human preference data, reward modeling, and policy optimization in model alignment.
References
Primary sources and free technical reading for RLHF.
Training Language Models to Follow Instructions with Human Feedback
The InstructGPT paper on demonstrations, preference models, PPO, evaluation, and safety behavior.
PaperFree
Learning to Summarize from Human Feedback
A foundational study of preference collection and reward-model-guided policy optimization.
PaperFree