Backpropagation
How neural networks learn using the chain rule.
For a small example, evaluate y = (2x + 1)^2 at the input x = 1. Run the forward values, propagate derivative 2u backward through u = 2x + 1, and verify with a small perturbation.
Backpropagation
In practice
A two-layer classifier refuses to learn. Inspect forward activations and gradient norms, compare autodiff with a numerical check, then isolate saturation or shape errors.
Initializing Interactive Playground...
Experiment notes
Record your experiment
Predict the sign of one weight update before the backward pass starts.
Implementation
Repair a failing Backpropagation starter
The starter deliberately follows a tempting but flawed assumption: Memorizing derivatives without graph reasoning
- Run the starter once and inspect the first wrong value, failed check, or incomplete branch.
- Repair the smallest relevant part of the code. Do not rewrite the whole solution.
- Check the solution, then explain why this correction works: Trace dependencies and accumulate vector-Jacobian products explicitly.
Check: The browser checks pass and you can name the failure mechanism before opening the explanation.
Failing starter lab
Check a chain-rule gradient
Differentiate a one-neuron squared-error loss and compare the result with finite differences.
Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.
Knowledge check
10 questions
Explanations after each answer · 80% to pass · Backpropagation
Chapter 1 of 10 · Core
The Computational Graph & Foundations of Reverse-Mode AD
At the core of deep learning optimization is the concept of a Computational Graph, a directed acyclic graph (DAG) where nodes represent mathematical operations (variables, tensors, or operators) and directed edges represent the flow of variables. Backpropagation is a highly specialized, computationally optimal implementation of reverse-mode Automatic Differentiation (AD) applied to these graphs.
To appreciate reverse-mode AD, we must compare it to forward-mode AD. Suppose we have a function .
- Forward-Mode AD computes the derivatives of all intermediate variables with respect to a single input variable in a single forward pass. To compute the full Jacobian , forward-mode requires separate passes, scaling as .
- Reverse-Mode AD computes the derivatives of a single output variable (the scalar loss , where ) with respect to all intermediate and input variables in a single backward pass. Its computational complexity scales as , which is for a scalar loss.
For a modern deep neural network with parameters and a scalar loss , naive forward-mode AD or numerical finite-difference approximation would require forward passes, making gradient computation mathematically impossible under realistic time constraints. Reverse-mode AD (backpropagation) allows us to compute the exact gradient with respect to *all* parameters in just one backward pass, taking roughly twice the time of a single forward pass.
Let a computational graph consist of vertices ordered such that represent the inputs, represent intermediate variables, and represents the final scalar loss. The forward pass computes: The backward pass computes the adjoint (or error signal) for each variable, defined as: Using the multivariate chain rule, we compute the adjoints in reverse order (): By caching the intermediate activations during the forward pass, we can compute the partial derivatives analytically during the backward pass without re-evaluating the operations, achieving linear time complexity .
Key equations
Video explanation
What Is Backpropagation Really Doing?
3Blue1Brown
Trace how output error assigns credit through a network before studying the full calculus.
References
Primary sources and free technical reading for Backpropagation.
Dive into Deep Learning
Free interactive book with executable implementations of neural networks, sequence models, CNNs, and Transformers.
BookFree
Automatic Differentiation in Machine Learning: a Survey
A rigorous survey of forward and reverse mode, computational graphs, and implementation tradeoffs.
PaperFree