LLM Tokenization
Understand how text becomes versioned token IDs, context usage, and model cost.
Quick start
Reach one useful result first
- Explain
- 15 min
- Visualize
- 10 min
- Practice
- 25 min
- Deep dive
- 25 min
Explanation
Start with the direct answer, follow one example, then check your understanding.
Choose your explanation depth
The mechanism stays the same; emphasis and evidence change.
A tokenizer breaks text into reusable pieces that a model can represent as numbers.
Carried example
“unbelievable” may become one token or several pieces depending on the tokenizer vocabulary.
Focus on the observable input, change, and output of tokenization.
Learning goals and prerequisites
After this lesson
- Explain why tokens are not words
- Inspect domain-specific token splits
- Version and test a tokenizer contract
Helpful before starting
- How LLMs generate text
- Basic familiarity with words and characters
5 chapters · about 1 hr 3 min estimated study
0/5 chapters marked complete
Direct answer
A tokenizer converts visible text into a sequence of integer IDs that a model can process. A token can be a whole word, part of a word, punctuation, whitespace, or a byte-like piece, so tokens are not the same thing as words or characters.
Follow the mechanism
- The tokenizer applies its configured normalization rules to the input text.
- It finds a sequence of known vocabulary pieces that represents the text.
- Each piece is replaced by its vocabulary ID and sent to the model as one position.
- Decoding uses the same vocabulary to turn IDs back into text.
Running example
Use the sentence “GraphRAG answers with cited evidence.” A transparent teaching tokenizer may expose Graph, RAG, spaces, ordinary words, and the final period as separate pieces. Another production tokenizer may split “GraphRAG” differently because it learned a different vocabulary. The sentence still looks like five words to a reader, yet the model may receive many more token positions. That difference affects context capacity, latency, and cost. It also explains why punctuation-heavy identifiers, source code, names, and multilingual text deserve their own tests.
What this does not mean
A token ID has no portable meaning by itself. ID 1842 can represent different pieces in different vocabularies, and cached IDs from one tokenizer must not be sent to a model that expects another. Word count and character count can help with rough planning, but only the deployed tokenizer can enforce the real request budget.
Ideas to understand
- Text → normalization → vocabulary pieces → token IDs is the tokenizer interface.
- Token boundaries depend on the exact tokenizer and vocabulary version.
- More pieces consume more context positions even when the visible meaning is unchanged.
- Budget and compatibility tests must use the tokenizer paired with the deployed model.
Decision and failure guide
Use LLM Tokenization with evidence
A complete explanation includes the conditions that make the technique useful and the nearby ways it can fail.
Before you call it ready
- Use the production tokenizer
- Test domain names and code
- Reserve output capacity
- Record tokenizer version
Common failure → better decision
- Equating words with tokens
- Measure actual token IDs.
- Reusing IDs across vocabularies
- Bind IDs to tokenizer version.
- Estimating cost from characters alone
- Count with the deployed implementation.
Was this lesson helpful?
Submit to the team when server feedback is available; otherwise this browser keeps a local copy and tells you so.
Concept connections
See prerequisites, applications, and nearby decisions.
Learn before
Often compared with
Continue next
Get one next lesson each week
A focused path reminder, not a marketing newsletter. Unsubscribe in one click.