intuitiveai

Stage 01 · UTF-8 bytes · CPU

Give text a stable representation.

Train byte-pair merges on training documents, freeze the rules, then encode and recover text you have never seen.

This small tokenizer is runnable, not GPT-2/tiktoken compatible. It needs Python 3.10+, no package install, account, API key or GPU. Stages 02–07 remain planned.

Before starting: Stage 00: prediction and held-out evidence. For background: tokenization concepts.

What you will demonstrate

  1. Trace overlapping pair counts and deterministic, non-overlapping byte-pair replacements.
  2. Implement pair counting, replacement and frozen merge application; preserve bytes through serialization and unseen text.
  3. Measure held-out representation costs without fitting on test text or confusing token units with language quality.

Count overlaps. Replace without overlap.

UTF-8 bytes for abab are [97, 98, 97, 98]. Count every adjacent pair within a document. Choose the most frequent pair, replace left-to-right, then recount.

(97, 98): 2 occurrences

(98, 97): 1 occurrence

Merge (97, 98) → 258

[97, 98, 97, 98] → [258, 258]

Now predict aaaa: its pair (97, 97) occurs three times, but replacements cannot share a byte. The result is two new tokens, not three. Counting frequency and applying a merge are different operations.

The exact training and encoding rules

Start with all 256 byte IDs. Reserve 256 for BOS and 257 for EOS; the first merge is 258. Break frequency ties by ascending numeric ID pairs. Never count a pair across document boundaries.

The supplied budget is 16 merges; stop early if no pair occurs twice. Encoding applies frozen pairs in rank order. Recounting on the input would create a different tokenizer.

No normalization, whitespace removal or regex pre-tokenization. Merges can cross spaces/newlines inside a document. This naive implementation is bounded to 128 merges and 256 KiB of training text, not designed for a large corpus.

Repair the non-overlapping merge.

Failing starter lab

Replace one pair without overlapping

Repair a single BPE merge step. Counting pairs, training a frozen tokenizer and interpreting held-out measurements belong to the downloadable project, not this component check.

Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.

Browser Python may load from a CDN. This component check is not the integrated project and is not saved to curriculum/cloud progress. Keep your downloaded report as evidence.

Train, serialize, then test the frozen rules.

Save these five files together. The corpus is the unchanged Stage 00 fixture: 8 training, 2 validation and 2 test documents. The downloaded project runs offline.

  1. Implement pair_counts, merge_pair and apply_merges in starter.py. The unfinished starter should fail.
  2. Run the tests, then build a report using your three functions.
  3. Inspect ordered merges, training IDs, corpus/tokenizer hashes, round trips and held-out representation costs.
  4. Reload the versioned tokenizer from the report, encode new text and explain a failure you repaired.
Run in the downloaded project folderbash
python3 -B test_tokenizer.py -v
python3 -B starter.py --output my-tokenizer-report.json
Reference and artifact checks—after your attempt
PyTorch Implementationbash
python3 -B test_tokenizer.py --reference -v
python3 -B tokenizer.py --output reference-report.json

The report contains the model under tokenizer. Restore it with Tokenizer.from_payload(report["tokenizer"]). Standalone save/load uses versioned JSON, never pickle; existing files are not overwritten.

Tests cover deterministic ties, document isolation, overlapping counts, rank order, all 256 bytes, unseen scripts, malformed artifacts and held-out fit isolation. A checksum identifies bytes and metadata; it is not an authenticity signature.

Lossless does not mean one token per character.

UTF-8 é is [195, 169]. A token may contain only part of that character. Concatenate token bytes before decoding; decoding each token separately can fail. Unseen scripts and emoji round-trip without UNK, but may use many tokens.

Precomposed and decomposed accents stay distinct. Literal <eos> stays ordinary text; only explicit framing inserts reserved EOS=257. The content decoder rejects control IDs. This boundary is not prompt-injection protection.

Compare measurements without changing their meaning

Count UTF-8 bytes, Unicode code points and content tokens separately. Report bytes/token and tokens/code point with boundaries excluded. This tiny English corpus is not evidence of multilingual efficiency.

The report refits raw-byte and BPE count baselines on training only, using fixed alpha 0.1 and one scored EOS per document. BOS is context-only. Their output alphabets and token units differ: raw token perplexity cannot rank these tokenizers.

Canonical-path nats per UTF-8 byte uses a common denominator and includes EOS loss. It is not normalized string likelihood: different token sequences can decode to the same bytes. Stage 00 used a different alphabet/UNK contract, so its scores are not substituted here.

Freeze the predeclared budget before looking at test evidence. Compression alone does not establish better prediction, reasoning or language understanding.

Predict an unfamiliar representation.

Add unseen scripts, code, emoji and literal special-token spellings only to held-out text. Predict byte counts and difficult boundaries, then verify round trips and unchanged learned merges. Explain why token perplexity cannot rank different tokenizers.

Artifact: A versioned JSON tokenizer/report with ordered merges, corpus and tokenizer hashes, split IDs, held-out ratios, baseline units and an explained unseen input.

A written transfer explanation still needs review. No mastery certificate is issued here, and existing learner records are not reinterpreted.

Next planned stage: build the data engine →

Primary background: subword BPE paper · Hugging Face BPE guide. Our variant uses its own document-wide byte policy.