Stage 00 · CPU · no dependencies

Make prediction measurable.

Fit a character-count model, then test whether it predicts held-out text better than a uniform baseline.

Stages 00–01 are runnable. Stages 02–07 remain a roadmap. This is not a neural GPT, and a component check does not complete the integrated project.

Prerequisites: Python and arrays · Probability and log loss. The downloaded project needs no account, API key, network call or GPU.

What you will demonstrate

Calculate mean negative log-likelihood in nats and convert it to perplexity.
Fit smoothed character-count baselines using training documents only.
Choose smoothing on validation, freeze it, and report test results without tuning on them.

Score the observed targets.

The correct next targets receive probabilities 0.5 and 0.25. Predict their loss before calculating. We score the actual targets, not whichever tokens rank first.

−ln(0.5) = 0.693147

−ln(0.25) = 1.386294

Mean NLL = (0.693147 + 1.386294) / 2 = 1.039721 nats

Perplexity = exp(1.039721) = 2.828427

Average logs over targets—not probabilities or equally weighted documents of different lengths. A uniform model over V targets has loss ln(V). Zero probability for an observed target gives infinite loss.

How does the count model predict?

A unigram counts targets without context. A bigram counts previous-character/next-character pairs. These count estimates need no gradient descent.

Add-alpha smoothing gives p(next | previous) = (pair count + alpha) / (previous count + alpha × V). An unseen context gives every target 1/V. Omitting alpha × V breaks normalization.

Each document starts with context-only BOS and ends with scored EOS. No transition joins documents. Only training text defines the alphabet; unseen characters become UNK.

First, repair the loss calculation.

Failing starter lab

Score observed next tokens

Complete the component calculation, then use the downloadable project to compare complete baselines. This browser check is practice, not completion of the integrated project.

Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.

Console

Run the code to inspect its output. Use Check solution only when you want to run the exercise checks.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

NLL: 1.039721 nats
Perplexity: 2.828427
ALL TESTS PASSED

Browser Python may load from a CDN. The downloaded project runs offline after saving the files. This component check is not saved to curriculum/cloud progress; keep your project report as evidence.

Build and evaluate the baseline.

Save all five files in one folder. Use Python 3.10 or later; install nothing. The original CC0 teaching corpus has 8 training, 2 validation and 2 test documents.

Read README.md and predict the two-target loss.
Complete mean_nll and conditional_probability in starter.py. The unfinished starter is expected to fail.
Run learner tests and generate a report using your repaired functions.
Inspect split IDs, corpus hash, alphabet, smoothing selection and score tables. Explain a failure before consulting the reference.

Run in the downloaded project folderbash

python3 test_stage00.py -v
python3 starter.py --output my-report.json

Only training text fits vocabulary and counts. Choose alpha using validation loss, freeze it, then report test results. Tuning after inspecting test results turns that set into development evidence. Existing report files are not overwritten.

Reference implementation and reproducibility checks

After your attempt, stage00.py provides the complete reference. It prints JSON or writes only to an explicitly requested new path.

PyTorch Implementationbash

python3 test_stage00.py --reference -v
python3 stage00.py --output reference-report.json

Tests check hand calculations, smoothing, boundaries, unknown characters, target weighting, split validation and test/selection isolation.

For corpus intuitiveai-stage00-v1, the reference selects alpha 0.1. Validation NLL is approximately 2.995732 for uniform, 2.763715 for unigram and 1.160772 for bigram. Test results are descriptive—not a tuning target.

The recurring artificial phrases are a debugging fixture, not a representative language benchmark. Exact cross-split duplicates are rejected; related phrases remain correlated. Lower loss here does not establish reasoning, factuality or general language quality.

Explain an unseen result.

Create new validation/test sentences without editing training documents. Predict which bigrams will be difficult, then report loss and unknown targets. Explain why you cannot tune on the old test and still call it untouched.

Insert Ω into only the test set. Why should unknown-target count change while vocabulary and chosen alpha stay fixed? What stronger split/deduplication checks would a real corpus need?

Artifact: A JSON report containing corpus hash, split IDs, train-only vocabulary, selection rule and all three baseline score tables.

The written explanation and independent transfer still need review. No mastery certificate is issued here.

Next: train a byte-level tokenizer →

Background: NLTK language-model documentation. Our implementation uses only the Python standard library.