Implementation sequence

A small GPT,
from scratch.

Text → tokens → attention → training → generation.

Eight stages with related lessons and checks. This is a build roadmap, not a complete runnable course.

Prerequisites: Python, arrays, and basic differentiation.

Explore language models

Stage 00
Make prediction measurable
Create a character-level bigram baseline and calculate negative log-likelihood by hand before using a neural network.
Probability and cross-entropy
Output & verification
Output
Baseline notebook + loss table
Check
Your held-out loss beats a uniform predictor and the split is reproducible.
Stage 01
Train the tokenizer
Implement byte-pair merges on a small corpus, freeze the vocabulary, then encode and decode real text, code, names, and multilingual samples.
LLM tokenization
Output & verification
Output
Versioned tokenizer + vocabulary
Check
Round trips are lossless and fertility is measured on held-out text.
Stage 02
Build the data engine
Clean and deduplicate a bounded corpus, create train and validation partitions, pack fixed-length sequences, and shift inputs into targets.
Data, evidence, and leakage
Output & verification
Output
Dataset manifest + batch inspector
Check
No document crosses the split and every target is the next input token.
Stage 03
Prove learning works
Write a small neural language model, trace tensor shapes, compare analytical gradients with finite differences, and overfit one tiny batch.
Backpropagation
Output & verification
Output
Training loop + gradient checks
Check
The tiny batch reaches near-zero loss and a broken gradient test fails loudly.
Stage 04
Implement causal attention
Implement query, key, value projections, scaled dot-product attention, a causal mask, multi-head attention, and one decoder block.
Transformer internals
Output & verification
Output
Attention module + mask tests
Check
Changing a future token cannot change an earlier position output.
Stage 05
Assemble a small GPT
Combine token and position representations, repeated decoder blocks, normalization, and a vocabulary projection with tied weights.
GPT architecture and pretraining
Output & verification
Output
Decoder-only model + parameter report
Check
Every tensor shape is asserted and parameter count matches the manual estimate.
Stage 06
Pretrain and evaluate
Add AdamW, warmup, decay, clipping, checkpoints, validation loss, perplexity, sample prompts, and controlled ablations.
Training systems and scaling
Output & verification
Output
Checkpoint + experiment report
Check
A held-out metric improves and the run can resume deterministically from a checkpoint.
Stage 07
Generate, package, and serve
Implement temperature and top-k sampling, protect context limits, measure latency, document limitations, and expose a small inference interface.
Model serving
Output & verification
Output
Sampling CLI + model card
Check
Fixed-seed output is reproducible and unsafe or unsupported uses are explicit.