# Stage 01 — train a byte-level BPE tokenizer

Version intuitiveai-byte-bpe-v1. Python 3.10+, standard library, CPU. No package,
GPU, model, account or provider key. Save five files together: this README,
starter.py, tokenizer.py, test_tokenizer.py and the unchanged Stage 00 corpus.json.
The website's corpus download intentionally points to that shared CC0 fixture.
Inside this repository only, a sibling-stage lookup also finds the corpus.

## Work it out first

UTF-8 bytes for `abab` are [97,98,97,98]. Adjacent pairs have counts (97,98):2
and (98,97):1. Merge (97,98) into ID 258: [258,258]. For `aaaa`, count (97,97)
THREE times but replace non-overlapping left-to-right: [258,258], not three new
tokens. Recount after every training step. A tie uses ascending numeric ID pairs.

All 256 bytes exist before training. IDs 256/257 are reserved BOS/EOS, and the
first learned merge gets ID 258. No UNK. No normalization or regex pre-splitting:
spaces/newlines are bytes, and merges may cross them within a document. Never
join separate documents for pair counting. This small, naive trainer is not
GPT-2/tiktoken compatible or designed for a large corpus.

## Build, then inspect

1. Complete pair_counts, merge_pair and apply_merges in starter.py. Do not mutate
   inputs. Encode with the frozen merge order, not new counts from the new text.
2. Run learner checks (unfinished functions are expected to fail):

   python3 -B test_tokenizer.py -v

3. Run the integrated report using YOUR three functions:

   python3 -B starter.py --output my-tokenizer-report.json

The report contains the versioned tokenizer, ordered merges, corpus/training IDs,
checksums, held-out representation statistics, transfer probes and count-baseline
tables. No script silently overwrites existing output. Omit --output to print.
Python's optional bytecode cache is suppressed by -B in these commands.

## Why bytes, not characters?

UTF-8 `é` is [195,169]. Either token alone may not decode as a character; join
all token bytes before decoding text. Unseen scripts/emoji still round-trip
because their bytes exist. That does NOT mean they get efficient tokenization.
`é` and `e` plus a combining accent remain different inputs: no normalization.
Literal `<eos>` is ordinary text; only explicit framing code inserts EOS=257.
The content decoder rejects BOS/EOS IDs. Lossless encoding is not prompt-injection
protection or evidence of semantic understanding.

## Read the measurements, not just the token count

The predeclared budget is 16 merges; stop early when every pair occurs fewer
than twice. Train only on labeled training documents. Never choose a new merge
budget based on test results and still call that test untouched. Full-corpus
hash changes when held-out evidence changes; learned merges should not change.

Counts report UTF-8 bytes, code points, content tokens, bytes/token and tokens/code
point. Boundaries are excluded from these counts. Do not describe these ratios
as universal word fertility or multilingual quality: this is a tiny English fixture.

Raw-byte and BPE uniform/unigram/bigram baselines are refitted using train only,
with fixed alpha=0.1, a context-only BOS and one scored EOS per document. Their
output vocabulary includes all content IDs and EOS, not BOS. Token perplexities
have DIFFERENT units and are not directly comparable. Canonical-path nats per
UTF-8 byte uses a common denominator (EOS loss included) but is NOT normalized
string likelihood: multiple legal segmentations can decode to the same bytes.
Do not infer better language modeling from compression. Stage 00 used another
alphabet/UNK contract; its scores are not silently reused here.

## Reference and serialization (after your attempt)

   python3 -B test_tokenizer.py --reference -v
   python3 -B tokenizer.py --output reference-report.json

The frozen model is report["tokenizer"]. In Python, restore it with
Tokenizer.from_payload(report["tokenizer"]) from the tokenizer module. You can
save/load a standalone model with model.save("new-tokenizer.json") and
Tokenizer.load("new-tokenizer.json"). save refuses overwrites. Canonical JSON
and SHA-256 identify exact metadata/order; hashes are not authenticity signatures.
Unknown versions, changed policies, invalid IDs/references and excessive expanded
payloads fail. No pickle. Budgets are 0–128, training/encoding at most 256 KiB,
JSON input at most 1 MiB. This favors readable algorithms, not speed or scale.

## Diagnose and transfer

- Predict the first merge for tied pairs, then reverse document order. Explain
  why merge decisions remain unchanged while document-order metadata can differ.
- Add only new test sentences containing `नमस्ते`, `東京`, emoji, code, decomposed
  accents or literal `<eos>`. Verify byte round-trips and frozen learned merges.
- Change a normalization policy: why must the format/version and evidence change?
- A round trip passes but one script uses more tokens/code point. What does it
  establish, what does it NOT establish, and what representative data is missing?
- Keep the JSON report plus a written explanation of units, splits, tie policy,
  a failed case you repaired and an unseen input. Automated checks are exercise
  evidence, not a reviewed mastery award. Stage 02's data engine remains planned.

## Provenance

Corpus: original Stage 00 teaching sentences, CC0-1.0 (corpus only). Source code
continues under the project's existing terms. No external corpus is downloaded.
Primary background: https://arxiv.org/abs/1508.07909 and the implementation guide
https://huggingface.co/learn/llm-course/en/chapter6/5 . This variant uses its own
document-wide byte policy, not the guide's word/pre-tokenization pipeline.