The attention mechanism that powers modern AI.
For a small example, in “The robot moved the glass because it cracked,” the token “it” needs earlier context. Project queries, keys, and values; compare the query for “it” with the allowed keys; normalize the scores; then trace how the weighted value vectors build its context.
A service routes policy documents by content. Tokenize, handle length and truncation, fine-tune or embed, evaluate difficult sections, and measure latency and memory.
Initializing Interactive Playground...
Before playing the steps, predict which earlier token “it” will weight most strongly. Then predict what must remain true after softmax.
The starter normalizes every score, including a future token that the query is not allowed to read. It also exponentiates raw values directly, so large scores can overflow.
Check: The browser checks pass and you can explain why masking happens before normalization and why raw scores, routing weights, and value vectors play different roles.
Failing starter lab
Apply an attention mask before a numerically stable softmax so forbidden keys receive exactly zero probability.
Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.
Explanations after each answer · 80% to pass · Transformers
Chapter 1 of 9 · Core
The core mechanism of the Transformer architecture (Vaswani et al., 2017) is Scaled Dot-Product Attention. Given query matrix , key matrix , and value matrix , attention is formulated as: where is the dimensionality of queries and keys.
To understand the necessity of the scaling factor , let's analyze the variance of the dot product before softmax.
Let and be query and key vectors representing a single row in and . Assume that the components and are independent random variables, each with mean 0 and variance 1: We assume that and are mutually independent for all indices .
The dot product is:
Using the linearity of expectation:
Since the elements are mutually independent: For two independent random variables and with mean 0: Thus: Summing across all dimensions:
As the query/key dimension grows large, the variance of the dot products scales linearly with . This means the dot products are highly likely to take on extremely large positive or negative values. Recall that the derivative of softmax is . When input logits have high variance, softmax can become very sharp, assigning probability near one to a maximum and near zero elsewhere. Many Jacobian entries then become small, weakening useful gradient signal; they are not mathematically all zero in finite precision-free analysis.
Dividing the dot product by scales the variance back to 1: This keeps the inputs in the active, high-gradient region of the softmax function, stabilizing optimization.
Key equations
Transformers, the Tech Behind LLMs
3Blue1Brown
See how tokens, embeddings, attention, and feed-forward blocks cooperate inside a transformer.
Primary sources and free technical reading for Transformers.