Embeddings & Context Windows
Learn how models represent text and how finite context changes application design.
Quick start
Reach one useful result first
- Explain
- 15 min
- Visualize
- 10 min
- Practice
- 25 min
- Deep dive
- 25 min
Explanation
Start with the direct answer, follow one example, then check your understanding.
Choose your explanation depth
The mechanism stays the same; emphasis and evidence change.
Embeddings turn tokens into learned coordinates; the context window limits what can be considered in one request.
Carried example
Two sentences about refunds can be close in embedding space even when they use different words.
Focus on the observable input, change, and output of embeddings and context.
Learning goals and prerequisites
After this lesson
- Distinguish token and retrieval embeddings
- Construct a complete token budget
- Choose long context or retrieval from evidence
Helpful before starting
- LLM tokenization
- Basic coordinate or similarity intuition
5 chapters · about 1 hr 3 min estimated study
0/5 chapters marked complete
Direct answer
An embedding represents an item as a learned vector so related items can be ranked by geometric similarity. A context window is the finite token budget available to one model request. Embeddings can help choose what enters that budget, but similarity and context capacity answer different questions.
Follow the mechanism
- Candidate passages are embedded and compared with the question.
- Authorization rules remove passages the user is not allowed to read.
- The application ranks the remaining passages and adds only those that fit after instructions, the question, and output capacity are reserved.
- The generation model receives the selected text as context and produces an answer; it does not see omitted passages.
Running example
Assume a teaching context limit of 18 tokens. Instructions, the question, and reserved output capacity use 7. A refund-policy passage costs 6 tokens with relevance 0.94, a returns FAQ costs 5 with relevance 0.82, and a shipping guide costs 5 with relevance 0.31. The first two exactly fill the budget: 7 + 6 + 5 = 18. The shipping guide is omitted even though it exists in the corpus. If the refund passage were unauthorized, it would need to be removed before ranking and budgeting rather than hidden after generation.
What this does not mean
High embedding similarity does not prove that a passage is true, authorized, current, or sufficient to support a claim. Likewise, a large advertised context window does not guarantee that the model will use every position equally well. Retrieval relevance, permissions, omissions, position effects, and answer support need separate tests.
Ideas to understand
- Question → similarity ranking → authorization → token budget → model context is the application chain.
- Embedding similarity is ranking evidence, not truth or permission.
- Every context plan must reserve capacity for fixed instructions, the user input, and output.
- Maximum context size and effective use of context must be measured separately.
Decision and failure guide
Use Embeddings & Context Windows with evidence
A complete explanation includes the conditions that make the technique useful and the nearby ways it can fail.
Before you call it ready
- Separate similarity from truth
- Reserve output tokens
- Preserve source identity
- Test effective use, not only maximum length
Common failure → better decision
- Calling similar vectors identical facts
- Treat similarity as ranking evidence only.
- Filling every available token
- Measure distraction, cost, and task quality.
- Truncating silently
- Expose omissions and test the policy.
Was this lesson helpful?
Submit to the team when server feedback is available; otherwise this browser keeps a local copy and tells you so.
Concept connections
See prerequisites, applications, and nearby decisions.
Learn before
Used by
Often compared with
Continue next
Get one next lesson each week
A focused path reminder, not a marketing newsletter. Unsubscribe in one click.