Back to Curriculum
Advanced•Modern AI
RLHF
Reinforcement Learning from Human Feedback.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
Large language models trained via self-supervised next-token prediction are highly capable text generators but lack any awareness of human values, truthfulness, or safety. They simply mimic the statistical distributions of their pre-training corpora—including toxic language, structural biases, and factual hallucinations.
The **Alignment Problem** asks: how can we guide a neural policy to be helpful, honest, and harmless?
### Supervised Fine-Tuning (SFT) and Its Limits:
A naive approach is to perform Supervised Fine-Tuning (SFT) using high-quality, human-curated demonstration pairs:
While SFT adapts the model's tone and formatting, it suffers from three fundamental limitations:
1. **Lack of Negative Feedback**: Cross-entropy only teaches the model what *to* generate. It cannot penalize toxic or hallucinated behaviors.
2. **Exposure Bias**: Because the model is trained with teacher forcing, error compound cascades quickly at inference when generating long text sequences.
3. **High Annotation Cost**: Writing high-quality, expert level answers is extremely slow and expensive.
**The Preference Scaling Insight**:
According to cognitive psychology, it is dramatically easier and faster for a human annotator to **rank and compare** two candidate model responses than to write the ideal response from scratch. RLHF exploits this comparative capability.
Key Equations
Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.