Grouped-Query Attention (GQA), Multi-Query (MQA), RoPE embeddings, and RMSNorm topology.
For a small example, one sentence contains subject, action, and location relationships. Let separate heads emphasize each relation, then concatenate their outputs and inspect why identical heads add little.
A document model must process 50,000 tokens. Compare full, local, sparse, and linear attention by receptive field, memory, accuracy, and implementation support.
Initializing Interactive Playground...
Predict how one control changes the Attention Variants & Norms result.
The starter deliberately follows a tempting but flawed assumption: Comparing variants with unequal parameter budgets
Check: The browser checks pass and you can name the failure mechanism before opening the explanation.
Failing starter lab
Grouped-query attention shares key/value heads across groups of query heads.
Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.
Explanations after each answer · 80% to pass · Attention Variants & Norms
Chapter 1 of 5 · Core
Standard Multi-Head Attention (MHA) in LLMs uses independent Query heads, Key heads, and Value heads where (e.g. 32 heads in Llama-2 7B).
This far exceeds single GPU capacity (80 GB A100), bottlenecking serving throughput.
Key equations
Attention in Transformers, Step by Step
3Blue1Brown
Walk through queries, keys, values, attention weights, and the movement of contextual information.
Primary sources and free technical reading for Attention Variants & Norms.