# Stage 00 — make next-token prediction measurable

Version: 2026-10-20-v1. Python 3.10+; standard library only. No package install,
GPU, account, provider key, database, network call, or downloaded dataset needed.
All five files belong in one folder: this README, corpus.json, starter.py,
stage00.py and test_stage00.py. Source files are UTF-8. Tests use tolerances for
floating-point logs; the corpus and splits are fixed, not randomized.

## Learn first

For observed target probabilities [0.5, 0.25], negative log-likelihood is
(-ln 0.5 - ln 0.25) / 2 = 1.039721 nats/target. Perplexity is exp(NLL) = 2.828427.
Average the log losses, not the raw probabilities. A zero probability for an
observed target gives infinite loss. All probabilities equal to one give zero.

A uniform baseline predicts 1/V. A unigram estimates target frequencies without
context. A bigram uses the previous character. Add-alpha smoothing computes
(count(previous,target)+alpha)/(count(previous)+alpha*V). Every possible target
contributes alpha to the denominator. An unseen context therefore gives a
uniform row. These are fitted count estimates, not a neural network or GPT.

## Your task

1. Predict the hand-calculated loss before running anything.
2. Complete the two functions in starter.py. Retain validation and edge cases.
3. Run learner checks (the incomplete starter is expected to fail):

   python3 test_stage00.py -v

4. Run the integrated pipeline with your functions:

   python3 starter.py --output my-report.json

The report path must be new; existing files are never overwritten. Omit --output
to print the report. The application writes only an explicitly requested report;
Python may also create its usual __pycache__ bytecode cache. Use python3 -B to
disable that cache if needed.

## Inspect the result

The report records the exact corpus SHA-256, document split IDs, vocabulary,
scoring contract, candidate smoothing values, validation selection and
train/validation/test losses for uniform, unigram and bigram models.

Only training documents fit the alphabet and counts. BOS is a context-only
marker; EOS is a scored target per document. Unknown held-out Unicode code
points map to UNK; they do not grow the vocabulary. There is no normalization.
No transition joins two documents. Loss is token-weighted, including EOS/UNK.

Predeclared alpha candidates are 0.1, 0.5, 1 and 2. Select using validation NLL,
freeze that choice and inspect test results only after selection. Test loss is
not used by the selection code. Rerunning deterministic code is allowed; tuning
after seeing test results makes that set development evidence. Create a fresh
protected set before a new final claim. Compare baselines on the SAME token,
boundary and split contract; cross-tokenizer perplexities are not comparable.

The supplied artificial corpus has 8 training, 2 validation and 2 test documents.
Its simple recurring phrases make it useful for debugging. Better validation
loss than uniform is a fixture sanity check, not proof of language competence.
Exact duplicate text crossing splits is rejected after whitespace/case folding;
this is not semantic or near-duplicate detection. Related phrases can remain
correlated, and that limits any generalization claim about this tiny corpus.

## Reference (after your attempt)

stage00.py is the complete reference engine and solution. To verify it:

   python3 test_stage00.py --reference -v
   python3 stage00.py --output reference-report.json

Tests cover hand loss, invalid probabilities, smoothing normalization,
document boundaries, unknown characters, token weighting, corpus validation,
train-only fitting and protected test/selection behavior. Passing these checks
is exercise evidence, not a tamper-resistant examination or independent review.

## Diagnose and transfer

- Explain why omitting alpha*V can make a conditional row fail to sum to one.
- Compare token weighting with an unweighted mean of per-document losses.
- Change only test text to include Ω. Check that vocabulary and selected alpha
  stay fixed while unknown-target count and test score can change.
- Try genuinely new sentences with the same frozen training set. Predict hard
  transitions before scoring and inspect which errors count models cannot fix.
- Do not call a copied training sentence a held-out example; both IDs and content
  matter. Document provenance and stronger deduplication for a real corpus.

## Provenance and limitations

The original miniature corpus is dedicated under CC0-1.0. See
https://creativecommons.org/publicdomain/zero/1.0/ . The source files are supplied
as part of the project's existing source-license terms; CC0 here identifies the
teaching CORPUS, not a new license for the entire repository.

This is a complete Stage 00, not the complete GPT course. Tokenizer training is
now available as a separate Stage 01 project. Neural updates, causal attention,
pretraining checkpoints and serving remain later stages. For background on n-gram models and smoothing see the official
NLTK language-model documentation: https://www.nltk.org/api/nltk.lm.html .