LLM decision guide
Tokens vs Embeddings
Tokens are discrete text units; embeddings are learned numeric representations used to compare or process meaning.
Choose Tokens when
- Count context and cost
- Inspect model input boundaries
Choose Embeddings when
- Measure semantic similarity
- Build dense retrieval
Decision table
| Dimension | Tokens | Embeddings |
|---|---|---|
| Representation | Integer IDs | Dense vectors |
| Primary job | Address vocabulary units | Encode learned features |
| Common failure | Unexpected splitting | Misleading similarity |
Example
A realistic choice
A tokenizer may split “GraphRAG” into pieces; an embedding can still place its sentence near related retrieval text.
Hands-on
Test the decision
Tokenize three domain terms, compare their splits, then rank short definitions by embedding similarity.