# Post-training mechanics lab v1

Python 3.10+, standard library only. Download all six files into the same folder.
No installation, account, model weights, network requests or GPU are required.
Fixtures are original synthetic examples; no participant or production data.

## Start with one topic

1. Read the matching lesson: RLHF, Quantization/LoRA, or GRPO.
2. Run `python3 test_labs.py --reference` (19 checks across the three topics).
3. Run `python3 run_lab.py rlhf --implementation reference` (or `lora`, `grpo`).
4. Implement your topic's functions in `starter.py`. It intentionally raises
   NotImplementedError; there is no hidden fallback to the reference.
5. Run `python3 test_labs.py --topic rlhf` (or `lora`, `grpo`).
6. Run `python3 run_lab.py rlhf --output my-rlhf-report.json`.
   Existing output files are never overwritten. Choose a new filename for each run.

The report records Python version, fixture SHA-256, configuration, implementation
and results. Compare reference and learner numeric results, ignoring only the
implementation label. Keep your code, report and a short explanation together.
No progress is uploaded by these scripts.

## RLHF: a real update in a finite model

Implement `dpo_loss` and `dpo_gradient`. A has logit theta, B has logit zero;
p(A)=sigmoid(theta). Reference is fixed [0.5,0.5]. The preferred pair is A over B.
We take five gradient steps with learning rate 0.2, using
L=softplus(-beta*theta). The general pair-loss function subtracts the reference
log-probability gap. At beta=1, the first step gives theta=0.1, p(A)=0.524979.

Try `--beta 0.5` and `--beta 2`. Record pair loss, KL and probability. A separate
transfer criterion declares B correct, deliberately conflicting with the training
labels. It is not used for parameter selection. Explain why improving label fit
can worsen correctness; this toy counterexample is not a population estimate.
For a real sequence model, collect completion-only summed log probabilities,
freeze reference parameters, group splits by prompt, and audit labeling quality.
This lab is DPO mechanics, not PPO, a neural reward model, or full LLM training.

## LoRA: forward, merge, quantize

Implement `lora_forward`, `merge`, `quantize`. W is out×in, A rank×in, B out×rank,
x a column vector. Matrices are guaranteed nonempty, compatible and finite in this
teaching API; a general-purpose layer must validate shapes/dtypes/devices itself.
Return fresh outputs, preserving the frozen base. Verify rectangular inputs,
zero B, and merged/unmerged parity. Never apply the adapter again after merging.

The runner supplies fixed adapters, not trained adapters. On an unseen input it
compares base, adapted, quantized base and quantized+adapted paths. MSE is relative
to matching unquantized outputs, not ground-truth task quality. Try `--bits 8`.
Quantization is signed symmetric narrow-range [-Q,Q], Q=2^(bits-1)-1;
scale=max(abs(weights))/Q, ties-to-even rounding, with scale=1 for all zeros.
It is not NF4. Codes remain Python integers; storage is not physically packed.
Extend with an outlier and per-block scales; keep transfer inputs out of any fitting.
You cannot infer GPU memory, latency or task quality from this script.

## GRPO: a loss audit, not a reasoning claim

Implement `advantages` and `surrogate`. Normalize per prompt using population
standard deviation. Equal rewards produce zeros. Ratio means current/old policy,
not current/reference. For each completion, maximize the mean of
min(ratio*A, clip(ratio,1-epsilon,1+epsilon)*A) over valid completion tokens.
Then average completions equally. epsilon=0.2; beta=0, so the KL term is explicitly
omitted. This convention is a declared teaching variant, not every GRPO trainer.
The reference chapter explains nonzero KL and the sampling-distribution caveat.

Try `--format-weight 0` and `--format-weight 3`. Correctness is exact integer
equality to 65; format flags are fixed fixture labels. Watch a wrong formatted
answer acquire positive advantage. No model outputs or generated programs execute.
Test an equal-reward transfer group, both clipping signs, variable lengths and
padding invariance. Report raw correctness separately from proxy reward.

## Evidence and next implementation boundary

Before running: predict a sign, magnitude or invariant. After running: record the
observed value, explain the mechanism and show one failure case. Add an unseen case
not copied from fixtures. Passing published tests is necessary but not independent
proof of mastery. There is no project auto-certification or hidden grading.

Next, port these invariants to a checkpoint-based experiment with a frozen model,
tokenizer and data revision. Measure a held-out task, peak device memory and latency
on named hardware. Do not compare differently normalized objectives as equivalent.
The GPT build path still has separate planned stages; these labs do not complete it.

## Primary sources

- DPO: https://arxiv.org/abs/2305.18290
- LoRA: https://arxiv.org/abs/2106.09685
- QLoRA: https://arxiv.org/abs/2305.14314
- GRPO / DeepSeekMath: https://arxiv.org/abs/2402.03300

Implementations and synthetic cases here are original teaching material, not
reproductions of the papers' training-scale results.