RLHF

Teach a preference, not just an answer

Pretraining learns to predict text. Supervised fine-tuning (SFT) learns from demonstrations. Preference training adds a different signal: for this prompt, response A was preferred to B. None of these objectives guarantees truth or safety.

A common RLHF pipeline fits a reward model to comparisons, then optimizes a policy with a reference penalty. DPO instead trains directly on paired preferences and reference log probabilities. We will implement one small DPO update, not a full PPO pipeline.

Before starting: be comfortable with natural logs, cross-entropy and a derivative. Use Reference for the policy objective and the PPO/DPO derivation.

Initializing Interactive Playground...

Work one pair by hand

Let A be preferred. The reference q assigns [0.5, 0.5]; the policy p assigns [0.8, 0.2]. At beta=1, the DPO margin is ln(0.8/0.5) − ln(0.2/0.5) = ln 4 = 1.386294. Its pair loss is −ln sigmoid(margin) = 0.223144 nats, down from ln 2 at p=q.

Parameterize p(A)=sigmoid(theta). With this uniform reference, margin=beta × theta, and the loss derivative is −beta × sigmoid(−beta × theta). At theta=0 and learning rate 0.2, one gradient step gives theta=0.1 and p(A)=0.524979. The downloadable runner computes this update and checks the derivative numerically.

For a real language model, use sums of completion-token log probabilities, excluding prompt and padding tokens, for the original sequence-level DPO objective. Do not substitute probabilities, average lengths silently, or update the reference checkpoint.

A lower loss can hide a worse answer

Suppose labels prefer a verbose answer even when its arithmetic is wrong. DPO can fit those labels successfully while task accuracy falls. Keep the training preference and the independent correctness check separate.

In Implementation, repair the stable loss, then run the two-response experiment. Freeze settings before evaluating the transfer fixture. In Practice, explain the disagreement between label fit and correctness; a loss number alone is not an acceptance test.

Experiment notes

Record your experiment

Keep beta=1 and move the policy probability of A from 0.5 to 0.8. Predict the direction of pair loss and exact KL, then explain why neither measures independent correctness.

PredictionWhat changed, and why?

Implementation

Fit a preference—and audit what it rewards

A two-response policy starts uniform. Training labels favor A; an independent transfer check favors B.

Repair and check the stable loss in the browser.
Implement the same function in starter.py and run the reference and learner experiments.
Compare before/after preference loss, exact KL and transfer correctness; explain the disagreement.

Check: Save a JSON report plus an explanation of the gradient, reference choice, and one proxy-reward failure.

Failing starter lab

Implement a stable DPO pair loss

These CPU labs isolate mechanisms. Passing the drill is not completion of the project, LLM training, or independent evidence of mastery.

Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.

Dataset: Authored finite preference fixtures v1

Console

Run the code to inspect its output. Use Check solution only when you want to run the exercise checks.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

ALL TESTS PASSED

Prerequisites: Next-token log probabilities · Cross-entropy and gradients · Train/validation/test splits

Build the complete CPU experiment

Download all six files into one folder. Python 3.10+; standard library only. No API key, GPU, account, model download or paid service. The browser exercise above is one component of this project.

Run the reference tests, then the reference experiment. Inspect the configuration, fixture hash, numerical results and limitations.
Implement the functions in starter.py, then run the learner tests and experiment. The unfinished starter intentionally fails; it never falls back to the answer.
Change one setting using the experiments below. Save a report with a new filename and add your prediction, result, explanation and limitation.

python3 test_labs.py --reference
python3 run_lab.py rlhf --implementation reference
python3 test_labs.py --topic rlhf
python3 run_lab.py rlhf --output my-rlhf-report.json

Experiment plan & debugging checks

Keep beta=1; take one step from theta=0.

Predict: p(A) rises above 0.5 and pair loss falls.

Inspect: theta=0.1, p(A)=0.524979; compare analytic and finite-difference gradients.

Use --beta 0.5 and --beta 2 with the same step count.

Predict: The margin scale and gradient both change; beta is not a learning-rate synonym.

Inspect: Report beta, final probability, pair loss, exact KL and transfer correctness.

Audit the held-out fixture whose correct answer is B.

Predict: Better preference fit can reduce independently checked correctness.

Inspect: Do not retune on this fixture; describe a new data collection/evaluation plan.

Next: port these invariants to a real model only after you can reproduce the small case. Freeze the checkpoint, tokenizer, split, objective, masks and evaluation before comparing runs. GPU training and independent learner validation remain separate work.

Where this fits in the GPT build path →

Back to Topic Library

Feedback & sharing

RLHF

Teach a preference, not just an answer

Work one pair by hand

A lower loss can hide a worse answer

Record your experiment

Implementation

Implement a stable DPO pair loss

Build the complete CPU experiment

Keep beta=1; take one step from theta=0.

Use --beta 0.5 and --beta 2 with the same step count.

Audit the held-out fixture whose correct answer is B.

Knowledge check

10 questions

Prove it on a new case

Demonstrations and preference feedback

Video explanation

References

Training Language Models to Follow Instructions with Human Feedback

Learning to Summarize from Human Feedback

Helpful?