Back to Curriculum
Basic•Statistics
Linear Algebra
Vectors, Matrices, Eigenvalues, and SVD.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
At the heart of linear algebra lies the abstract algebraic structure of a **Vector Space**. Formally, a vector space over a field (in machine learning, almost exclusively the real numbers ) is a set equipped with two binary operations—vector addition () and scalar multiplication ()—that satisfy the following eight axioms for all and :
1. **Associativity of Addition**:
2. **Commutativity of Addition**:
3. **Identity Element of Addition**: There exists an element such that
4. **Inverse Elements of Addition**: For every , there exists an element such that
5. **Compatibility of Scalar Multiplication**:
6. **Identity Element of Scalar Multiplication**: , where is the multiplicative identity of
7. **Distributivity of Scalar Multiplication over Vector Addition**:
8. **Distributivity of Scalar Addition over Scalar Multiplication**:
### Linear Independence, Span, and Bases
Let be a set of vectors.
* The **span** of , denoted as , is the set of all linear combinations of the vectors in :
* The set is **linearly independent** if the only trivial representation of the zero vector is the one with all zero coefficients:
If there exists a set of coefficients not all zero that sums to , the vectors are **linearly dependent**, meaning at least one vector can be expressed as a linear combination of the others.
* A **basis** of a vector space is a linearly independent set of vectors that spans . Every vector can be uniquely expressed as a linear combination of the basis vectors. To prove uniqueness, suppose there are two representations of in terms of basis vectors :
Since the basis vectors are linearly independent, we must have for all , meaning .
* The **dimension** of , denoted , is the cardinality of its basis. This dimension is invariant to the choice of basis (by the Dimension Theorem).
### Linear Transformations
A mapping between two vector spaces over the same field is a **linear transformation** if it preserves vector addition and scalar multiplication:
Every linear transformation between finite-dimensional spaces can be represented as a matrix. Specifically, if is a basis for and is a basis for , the matrix representing has columns containing the coordinates of relative to the basis .
### Machine Learning Connection: High-Dimensional Geometry
In machine learning, data points are typically represented as high-dimensional feature vectors residing in . The geometric properties of change dramatically as increases—a phenomenon known as the **curse of dimensionality**:
1. **Volume Concentration**: The volume of a -dimensional hypersphere of radius scales as , meaning almost all of its volume is concentrated in a thin outer shell near the boundary.
2. **Sparsity**: The volume of the bounding hypercube grows exponentially, causing data points to become extremely sparse.
3. **Equidistance**: The Euclidean distance between any two randomly selected points in a high-dimensional space converges to the same value, making distance-based algorithms (like -Nearest Neighbors) highly sensitive to noise.
Dimensionality reduction techniques (like PCA, Autoencoders, and t-SNE) exploit the **manifold hypothesis**, which asserts that high-dimensional real-world data concentrates near low-dimensional, non-linear sub-manifolds embedded within .
Key Equations
Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.