Linear Algebra
Vectors, Matrices, Eigenvalues, and SVD.
For a small example, move the point (1, 0) by a 90-degree rotation matrix. Multiply matrix by vector, track dimensions, and interpret the result geometrically rather than as symbol pushing.
Linear Algebra
In practice
A dataset has many redundant sensor readings. Center the matrix, inspect singular values, project to principal directions, and measure retained variance.
Initializing Interactive Playground...
Experiment notes
Record your experiment
Predict which direction a basis vector will move and whether area will grow, shrink, or flip.
Implementation
Repair a failing Linear Algebra starter
The starter deliberately follows a tempting but flawed assumption: Treating matrices as tables without geometry
- Run the starter once and inspect the first wrong value, failed check, or incomplete branch.
- Repair the smallest relevant part of the code. Do not rewrite the whole solution.
- Check the solution, then explain why this correction works: Connect every operation to a transformation, projection, or bilinear form.
Check: The browser checks pass and you can name the failure mechanism before opening the explanation.
Failing starter lab
Project one vector onto another
Implement vector projection and verify that the residual is orthogonal to the target direction.
Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.
Knowledge check
10 questions
Explanations after each answer · 80% to pass · Linear Algebra
Chapter 1 of 7 · Core
Vector Spaces, Bases, and Dimension
At the heart of linear algebra lies the abstract algebraic structure of a Vector Space. Formally, a vector space over a field (in machine learning, almost exclusively the real numbers ) is a set equipped with two binary operations—vector addition () and scalar multiplication ()—that satisfy the following eight axioms for all and :
- Associativity of Addition:
- Commutativity of Addition:
- Identity Element of Addition: There exists an element such that
- Inverse Elements of Addition: For every , there exists an element such that
- Compatibility of Scalar Multiplication:
- Identity Element of Scalar Multiplication: , where is the multiplicative identity of
- Distributivity of Scalar Multiplication over Vector Addition:
- Distributivity of Scalar Addition over Scalar Multiplication:
Linear Independence, Span, and Bases
Let be a set of vectors.
- The span of , denoted as , is the set of all linear combinations of the vectors in :
- The set is linearly independent if the only trivial representation of the zero vector is the one with all zero coefficients:
If there exists a set of coefficients not all zero that sums to , the vectors are linearly dependent, meaning at least one vector can be expressed as a linear combination of the others.
- A basis of a vector space is a linearly independent set of vectors that spans . Every vector can be uniquely expressed as a linear combination of the basis vectors. To prove uniqueness, suppose there are two representations of in terms of basis vectors :
Since the basis vectors are linearly independent, we must have for all , meaning .
- The dimension of , denoted , is the cardinality of its basis. This dimension is invariant to the choice of basis (by the Dimension Theorem).
Linear Transformations
A mapping between two vector spaces over the same field is a linear transformation if it preserves vector addition and scalar multiplication:
Every linear transformation between finite-dimensional spaces can be represented as a matrix. Specifically, if is a basis for and is a basis for , the matrix representing has columns containing the coordinates of relative to the basis .
Machine Learning Connection: High-Dimensional Geometry
In machine learning, data points are typically represented as high-dimensional feature vectors residing in . The geometric properties of change dramatically as increases—a phenomenon known as the curse of dimensionality:
- Volume Concentration: The volume of a -dimensional hypersphere of radius scales as , meaning almost all of its volume is concentrated in a thin outer shell near the boundary.
- Sparsity: The volume of the bounding hypercube grows exponentially, causing data points to become extremely sparse.
- Equidistance: The Euclidean distance between any two randomly selected points in a high-dimensional space converges to the same value, making distance-based algorithms (like -Nearest Neighbors) highly sensitive to noise.
Dimensionality reduction techniques (like PCA, Autoencoders, and t-SNE) exploit the manifold hypothesis, which asserts that high-dimensional real-world data concentrates near low-dimensional, non-linear sub-manifolds embedded within .
Key equations
Video explanation
Vectors: What Even Are They?
3Blue1Brown
Build the geometric and computational meaning of vectors used throughout machine learning.
References
Primary sources and free technical reading for Linear Algebra.