Large language models

Predict one token at a time.

A language model assigns a score to each possible next token. Softmax turns those scores into probabilities.

Sampling selects a token from that distribution. The token is appended to the context, and the model predicts again.

pᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)

Move the temperature slider. How does the probability of “mat” change?

A five-token vocabulary01 / 03

The cat sat on the ____

mat50.7%

floor25.2%

chair13.8%

roof6.2%

table4.2%

Temperature 1.0

Lower temperature concentrates probability on high-scoring tokens.

Illustrative scores, not a live language model. Sampling and temperature scaling are computed in your browser.

Experiment notes

Record your experiment

Predict how temperature will change the next-token probabilities.

PredictionWhat changed, and why?

Implementation

Implement a next-token update

Five logits start equal. Update them so the observed token becomes more likely.

Compute softmax probabilities.
Subtract learning_rate × (probability − target).
Verify that target probability rises and cross-entropy falls.

Check: The checks pass for every target position, including nonzero initial logits.

Failing starter lab

Cross-entropy and gradient descent

Complete one gradient update on five logits. This toy model isolates the loss used in next-token pretraining; it is not a transformer.

Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.

Console

Run the code to inspect its output. Use Check solution only when you want to run the exercise checks.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

Loss: 1.609 -> 0.396
ALL TESTS PASSED

Chapter 1 of 8 · Core

Tokenization: From Raw Text to Model Inputs

Large Language Models do not operate on raw characters or whole words — they operate on tokens, sub-word units derived through data-driven segmentation algorithms. The choice of tokenizer profoundly affects model vocabulary size, sequence length, and downstream performance.

Byte-Pair Encoding (BPE): Originally a data compression algorithm (Gage, 1994), BPE was adapted for NLP by Sennrich et al. (2016). The algorithm starts with a vocabulary of individual characters (or bytes) and iteratively merges the most frequent adjacent pair of tokens into a single new token. After k merge operations, the vocabulary size is |V_base| + k. GPT-2 and GPT-3 use byte-level BPE, starting from 256 byte values rather than Unicode characters, guaranteeing that any input can be tokenized without unknown tokens.

WordPiece: Developed by Schuster & Nakajima (2012) and used in BERT, WordPiece is similar to BPE but selects merges that maximize the likelihood of the training corpus under a unigram language model. The merge criterion is: choose the pair (a, b) that maximizes P(ab) / (P(a) · P(b)), i.e., the mutual information between adjacent tokens.

SentencePiece: Proposed by Kudo & Richardson (2018), SentencePiece treats the input as a raw byte stream and applies either BPE or a unigram language model. The unigram model assigns a probability to each sub-word candidate and finds the segmentation that maximizes the total log-likelihood via the Viterbi algorithm. SentencePiece is language-agnostic and does not require pre-tokenization (e.g., whitespace splitting), making it ideal for multilingual models like T5 and LLaMA.

The tokenizer defines the model's "perception" of language. A vocabulary of ~32K–100K tokens balances compression efficiency (fewer tokens per sentence) against embedding table size. Fertility — the average number of tokens per word — is a key metric: lower fertility means more efficient processing. Languages underrepresented in training data often have higher fertility, leading to performance disparities.

Key equations

NextFrom Static Word Embeddings to Contextual Representations

Video explanation

Large Language Models Explained Briefly

3Blue1Brown

Connect next-token prediction to the behavior and limitations learners observe in language models.

Watch on YouTube

References

Primary sources and free technical reading for LLMs.

Language Models are Few-Shot Learners

The GPT-3 paper on scale, in-context learning, evaluation, and limitations.

PaperFree

Hugging Face LLM Course

A free practical course on tokenizers, Transformers, fine-tuning, datasets, and model sharing.

CourseFree

Back to Topic Library

Feedback & sharing