How Linear Algebra Powers Machine Learning (ML)
Fangfang Lee from IBM explains how linear algebra is the mathematical foundation of machine learning, enabling computers to understand data. The summary covers key concepts like vectorization, similarity metrics (Euclidean distance, cosine similarity), and dimensionality reduction using Singular Value Decomposition (SVD).
Linear algebra provides the essential mathematical framework for machine learning, translating complex data like images, text, and audio into a numerical language that computers can process. It allows us to represent data in high-dimensional spaces where relationships and patterns can be measured, compared, and learned.
From Raw Data to Mathematical Objects
The first step in making data computable is factorization—converting it into the fundamental objects of linear algebra. This process, often called vectorization, prepares the data for model training.
- Scalar: The simplest unit, a single number representing a magnitude (e.g., 5, 2.6, or π).
- Vector: A one-dimensional array or list of numbers. For example, a piece of text can be tokenized and represented as a vector (or embedding) that captures its semantic meaning.
- Matrix: A two-dimensional structure of rows and columns. An image, for instance, can be represented as a matrix where each entry corresponds to a pixel’s intensity.
- Tensor: A generalization of these concepts to higher dimensions (3D or more), commonly used in frameworks like TensorFlow and PyTorch.
These frameworks have highly optimized vector and matrix operations, allowing for computation at a massive scale.
Measuring Relationships: Vector Similarity
Once data is in vector form, we can mathematically compare different data points. For example, we can determine how similar two sentences are by calculating the distance between their respective vectors.
Euclidean Distance
Euclidean distance is a straightforward metric that computes the point-wise similarity between two vectors. It is calculated as the square root of the sum of the squared differences between corresponding elements. The result is an unbounded scalar that indicates the magnitude of their similarity—a smaller number means the vectors are closer or more similar.
distance = sqrt(Σ(A_i - B_i)²)
Cosine Similarity
Cosine similarity measures the cosine of the angle (θ) between two vectors, focusing on their orientation rather than their magnitude. This is particularly useful for high-dimensional data like text embeddings.
It is calculated by taking the dot product of the two vectors and dividing it by the product of their magnitudes.
similarity = cos(θ) = (A ⋅ B) / (||A|| * ||B||)
The output is a standardized metric bounded between -1 and 1:
- 1: The vectors point in the exact same direction, indicating maximum similarity.
- 0: The vectors are orthogonal (at a 90-degree angle), meaning the features they represent are independent.
- -1: The vectors point in opposite directions, indicating maximum dissimilarity.
The Engine of Neural Networks
The vector dot product, used in cosine similarity, extends to the matrix dot product, which is one of the most critical and frequently used operations in machine learning. It is fundamental to all types of neural networks, from simple architectures to state-of-the-art transformer models.
Taming High-Dimensional Data with SVD
Modern models are trained on billions of data points (tokens), making computations across all dimensions impractical. Dimensionality reduction techniques are essential for making this process manageable. Singular Value Decomposition (SVD) is a powerful matrix factorization method that breaks down a complex, large matrix into smaller, more interpretable components.
Given a large matrix A (e.g., a dataset of user movie ratings), SVD decomposes it into three smaller matrices: A = UΣVᵀ.
- U (Left Singular Matrix): An orthogonal matrix representing features in the row space. In the example, this captures user features and behaviors.
- Σ (Sigma): A diagonal matrix containing singular values, ordered from largest to smallest. These values signify the importance of each feature dimension.
- Vᵀ (Transposed Right Singular Matrix): An orthogonal matrix representing features in the column space. In the example, this captures information and features about the movies.
By using SVD, we can analyze the singular values in Σ and choose to keep only the most significant features, effectively reducing the dimensionality of the data while retaining the most informative structures. SVD is exceptionally versatile as it can be applied to any matrix, regardless of its shape.
Conclusion: The Cornerstone of Machine Learning
Linear algebra is not just a prerequisite; it is the operational core of machine learning. It provides the tools to:
- Represent raw data as structured numerical objects.
- Compute relationships and transformations efficiently through matrix operations.
- Optimize and refine these representations through methods like SVD.
Ultimately, linear algebra is what turns raw data into mathematical form, computations into organized structures, and those structures into the actionable intelligence that powers AI.