LLM Decoding & Sampling
Compare greedy decoding, temperature, and top-p through visible probability changes.
Quick start
Reach one useful result first
- Explain
- 15 min
- Visualize
- 10 min
- Practice
- 25 min
- Deep dive
- 25 min
Explanation
Start with the direct answer, follow one example, then check your understanding.
Choose your explanation depth
The mechanism stays the same; emphasis and evidence change.
Decoding changes how the model chooses among plausible next tokens.
Carried example
Low temperature concentrates probability on the highest-scoring token; higher temperature makes alternatives more likely.
Focus on the observable input, change, and output of decoding.
Learning goals and prerequisites
After this lesson
- Compare greedy, temperature, and top-p decoding
- Choose settings from task requirements
- Build repeatable decoding tests
Helpful before starting
- How LLMs generate text
- Basic probability intuition
5 chapters · about 1 hr 3 min estimated study
0/5 chapters marked complete
Direct answer
The model produces raw next-token scores; decoding decides how those scores become the next visible token. Greedy decoding always chooses the largest probability, while sampling allows controlled variation among plausible candidates.
Follow the mechanism
- The model produces one logit for every vocabulary token.
- Temperature rescales the logits before softmax. Lower values magnify score differences; higher values compress them.
- Top-p keeps the smallest high-probability candidate set whose cumulative mass reaches the chosen threshold.
- The decoder selects or samples a token from the allowed set, appends it, and repeats with a new distribution.
Running example
For “The capital of France is”, use logits Paris: 3.8, Lyon: 1.5, and France: 0.8. At temperature 1, softmax gives approximately 87%, 9%, and 4%. At temperature 0.5, Paris rises to about 99%; at temperature 2, it falls to about 65% while Lyon and France become more likely. The model has not learned or forgotten anything between these settings. Only the selection distribution changed. With top-p at 0.90, the permitted set also changes as the temperature changes because cumulative probabilities are recalculated at every step.
What this does not mean
Higher temperature does not make a model more knowledgeable, and lower temperature does not make an answer correct. For a machine-consumed label or tool call, a closed schema and validator provide a stronger guarantee than temperature. For brainstorming, diversity can be useful, but it should still be measured against task quality and safety.
Ideas to understand
- Logits → temperature → softmax → top-p set → selection is the decoding chain.
- Temperature changes probability concentration; it does not add knowledge.
- Top-p creates a different candidate set for each next-token distribution.
- Choose decoding for the task, then enforce validity with explicit application controls.
Decision and failure guide
Use LLM Decoding & Sampling with evidence
A complete explanation includes the conditions that make the technique useful and the nearby ways it can fail.
Before you call it ready
- Match decoding to task
- Validate machine-consumed output
- Record every parameter
- Retest after model changes
Common failure → better decision
- Using high temperature to improve knowledge
- Temperature only reshapes choices from existing scores.
- Calling low temperature deterministic correctness
- Keep validation and evaluations.
- Changing several controls at once
- Run controlled comparisons.
Was this lesson helpful?
Submit to the team when server feedback is available; otherwise this browser keeps a local copy and tells you so.
Concept connections
See prerequisites, applications, and nearby decisions.
Learn before
Continue next
Get one next lesson each week
A focused path reminder, not a marketing newsletter. Unsubscribe in one click.