Neural Networks
The building blocks of deep learning: Perceptrons and MLPs.
For a small example, two inputs feed a weighted sum and ReLU. Calculate the pre-activation, apply ReLU, and change one weight to see when the neuron switches on.
Neural Networks
In practice
A small multilayer network learns from pixel vectors. Normalize inputs, choose hidden capacity, monitor train and validation loss, and inspect confidently wrong digits.
Initializing Interactive Playground...
Experiment notes
Record your experiment
Choose one input and predict which downstream activation should strengthen or weaken.
Implementation
Repair a failing Neural Networks starter
The starter deliberately follows a tempting but flawed assumption: Adding depth before validating the pipeline
- Run the starter once and inspect the first wrong value, failed check, or incomplete branch.
- Repair the smallest relevant part of the code. Do not rewrite the whole solution.
- Check the solution, then explain why this correction works: Overfit a tiny batch and beat a simple baseline first.
Check: The browser checks pass and you can name the failure mechanism before opening the explanation.
Failing starter lab
Implement a two-layer forward pass
Compute hidden ReLU activations and output probabilities with explicit matrix shapes.
Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.
Knowledge check
10 questions
Explanations after each answer · 80% to pass · Neural Networks
Chapter 1 of 5 · Core
The Perceptron, Linear Separability, and Convergence Proof
The Artificial Perceptron, formulated by Frank Rosenblatt (1958), is the historical foundation of artificial neural networks. A perceptron is a binary classifier mapping an input vector to a scalar output via an affine transformation followed by a step function: Without loss of generality, we can absorb the bias into the weight vector and augment the input vector with a constant , writing the decision rule as .
The Perceptron Learning Algorithm is an online update rule: for each training observation , if the current weight vector misclassifies the point (meaning ), we update the weights: where is the learning rate.
Proof of the Perceptron Convergence Theorem (Novikoff, 1962):
Let the training dataset be linearly separable. This assumption guarantees the existence of at least one separating direction. Choose any unit-normalized separator with positive margin such that: where represents the functional margin, and we normalize . Let be the maximum Euclidean norm of the input vectors.
Let the initial weights be . Suppose the algorithm makes its -th mistake on observation , updating from to with :
- Lower Bound on :
Taking the inner product of both sides with : Using the separating margin condition : By induction, starting from : By the Cauchy-Schwarz inequality, since :
- Upper Bound on :
Computing the squared norm of : Since a mistake was made, the term . Furthermore, : By induction, starting from :
- Combining the Bounds:
Combining the lower and upper bounds on : This elegant proof establishes that the perceptron is guaranteed to converge to a separating hyperplane in at most steps, depending entirely on the ratio of data spread to the margin.
However, in 1969, Minsky and Papert proved that a single-layer perceptron cannot represent the XOR function because XOR is not linearly separable, inducing a structural bottleneck that was only resolved by multi-layer networks.
Key equations
Video explanation
But What Is a Neural Network?
3Blue1Brown
Build an intuition for layers, activations, weights, and learned representations from a concrete example.
References
Primary sources and free technical reading for Neural Networks.