Linear Algebra Basics: Vectors, Matrices, Transformations

Linear algebra is the mathematics of structured collections — vectors and matrices — and the transformations between them. Where calculus deals with continuously changing quantities, linear algebra deals with structured systems. The two are deeply connected, and modern applied mathematics relies on both.

The single most important thing to understand about linear algebra is that matrices represent linear transformations. A matrix is not just a grid of numbers — it is a function that takes vectors as input and produces vectors as output. Once this is internalized, the rest of the subject opens up.

This article covers what linear algebra is really about, vectors (more than arrows), linear transformations, matrices as representations of transformations, the operations on matrices, determinants and what they measure, linear independence and span, vector spaces, and the connections to machine learning, computer graphics, and physics.

What linear algebra is really about

The historical motivation for linear algebra was solving systems of linear equations. A system like

$2x+3y=7$ $x-y=1$

has a unique solution: $x=2,y=1$. For two equations in two unknowns, you can solve by hand. For many equations in many unknowns — the case in real applications — you need systematic methods.

These methods are linear algebra. The system above can be written

$\left(\begin{array}{cc}2& 3\\ 1& -1\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}7\\ 1\end{array}\right)$

or, more compactly, $A\mathbf{x}=\mathbf{b}$. Solving the system is finding the vector $\mathbf{x}$ such that, when the matrix $A$ acts on $\mathbf{x}$, the result is $\mathbf{b}$.

This formulation generalizes. Instead of two equations in two unknowns, you can have 1,000 equations in 1,000 unknowns, or 1,000,000 in 1,000,000. The algebra is the same; only the size changes.

Modern linear algebra is far broader than the original motivation. It includes vector spaces, linear transformations, inner products, eigenvalues, decompositions, and many other concepts. But the original motivation — systematic methods for structured linear problems — runs through everything.

Vectors: more than arrows

In a first encounter, vectors are introduced as arrows in space — quantities with magnitude and direction. This is a useful first picture, but it is too narrow.

A vector is an element of a vector space. That phrase is circular until you know what a vector space is. Operationally: a vector is a structured collection of numbers (its components) that obeys particular rules about addition and scaling.

In two dimensions, a vector is a pair: $\mathbf{v}=(3,4)$. In three dimensions, a triple: $\mathbf{v}=(3,4,5)$. In $n$ dimensions, an $n$-tuple. The number of components is the dimension of the vector.

You can add two vectors of the same dimension componentwise:

$\mathbf{u}+\mathbf{v}=(u_1,u_2)+(v_1,v_2)=(u_1+v_1,u_2+v_2)$

You can multiply a vector by a scalar (a number):

$c\mathbf{v}=c(v_1,v_2)=(c{v}_1,c{v}_2)$

These two operations — vector addition and scalar multiplication — are the defining operations.

Why "more than arrows"? Because vectors can represent many things that are not spatial:

• A list of features describing a data point (height, weight, age) is a vector.
• The probability distribution over $n$ outcomes is a vector.
• A function evaluated at $n$ points is a vector.
• The polynomial $3+2x+x^2$ can be encoded as the vector $(3,2,1)$.

The geometric picture (arrows) is useful for two and three dimensions. For higher dimensions and abstract applications, the algebraic picture (tuples of numbers) is more general.

Vector operations

The basic operations on vectors:

Magnitude (length):

$\Vert \mathbf{v}\Vert =\sqrt{v_1^2+v_2^2+\cdots +v_n^2}$

For $\mathbf{v}=(3,4)$: $\Vert \mathbf{v}\Vert =\sqrt{9+16}=5$.

Unit vector:

$\hat{\mathbf{v}}=\frac{\mathbf{v}}{\Vert \mathbf{v}\Vert }$

A vector with magnitude 1 pointing in the same direction as $\mathbf{v}$.

Dot product (or inner product):

$\mathbf{u}\cdot \mathbf{v}=u_1{v}_1+u_2{v}_2+\cdots +u_n{v}_n$

The dot product produces a scalar. Geometrically, $\mathbf{u}\cdot \mathbf{v}=\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert \cos \theta$, where $\theta$ is the angle between the vectors. The dot product measures how much two vectors "point in the same direction."

Special cases:

• $\mathbf{u}\cdot \mathbf{v}=0$ means the vectors are orthogonal (perpendicular).
• $\mathbf{u}\cdot \mathbf{u}=\Vert \mathbf{u}{\Vert }^2$.

Example 1: For $\mathbf{u}=(1,2,3)$ and $\mathbf{v}=(4,5,6)$:

$\mathbf{u}\cdot \mathbf{v}=1\cdot 4+2\cdot 5+3\cdot 6=4+10+18=32$

Cross product (only in 3D):

$\mathbf{u}\times \mathbf{v}=(u_2{v}_3-u_3{v}_2,u_3{v}_1-u_1{v}_3,u_1{v}_2-u_2{v}_1)$

Produces a vector perpendicular to both $\mathbf{u}$ and $\mathbf{v}$. Used heavily in physics (torque, magnetic forces) and computer graphics (normal vectors).

Linear transformations

A linear transformation is a function $T$ from one vector space to another that preserves vector addition and scalar multiplication:

$T(\mathbf{u}+\mathbf{v})=T(\mathbf{u})+T(\mathbf{v})$ $T(c\mathbf{u})=c\cdot T(\mathbf{u})$

In English: if you add two vectors and then apply $T$, you get the same result as applying $T$ to each and adding the results. If you scale a vector and then apply $T$, you get the same result as applying $T$ first and then scaling.

Linear transformations have a profound consequence: a linear transformation is completely determined by what it does to a basis. If you know how $T$ transforms each basis vector, you know how it transforms every vector — because every vector is a linear combination of basis vectors, and $T$ respects linear combinations.

This is why matrices work. A matrix is a compact way to record what a linear transformation does to each basis vector.

Examples of linear transformations:

• Rotation in 2D: $T(x,y)=(x\cos \theta -y\sin \theta ,x\sin \theta +y\cos \theta )$.
• Reflection: $T(x,y)=(x,-y)$ reflects across the x-axis.
• Scaling: $T(x,y)=(cx,cy)$ scales both components by $c$.
• Projection: $T(x,y)=(x,0)$ projects onto the x-axis.

Each can be represented as a matrix.

Matrices: representing transformations

A matrix is a rectangular grid of numbers. We write a matrix with $m$ rows and $n$ columns as

$A=\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mn}\end{array}\right)$

Each entry $a_{ij}$ is the entry in row $i$, column $j$.

Matrix-vector multiplication. If $A$ is an $m\times n$ matrix and $\mathbf{x}$ is an $n$-dimensional column vector, then $A\mathbf{x}$ is an $m$-dimensional column vector defined by:

$(A\mathbf{x}{)}_i={\sum }_{j=1}^n{a}_{ij}{x}_j$

In words: the $i$th entry of $A\mathbf{x}$ is the dot product of the $i$th row of $A$ with $\mathbf{x}$.

Example 2: $A=\left(\begin{array}{cc}1& 2\\ 3& 4\\ 5& 6\end{array}\right)$, $\mathbf{x}=\left(\begin{array}{c}7\\ 8\end{array}\right)$.

$A\mathbf{x}=\left(\begin{array}{c}1\cdot 7+2\cdot 8\\ 3\cdot 7+4\cdot 8\\ 5\cdot 7+6\cdot 8\end{array}\right)=\left(\begin{array}{c}23\\ 53\\ 83\end{array}\right)$

A 3×2 matrix maps a 2-dimensional vector to a 3-dimensional vector. The dimensions are essential — the matrix must have as many columns as the vector has components.

The connection to linear transformations. The columns of $A$ are exactly the images of the basis vectors. The first column $(1,3,5)$ is $A{\mathbf{e}}_1$ where ${\mathbf{e}}_1=(1,0)$. The second column $(2,4,6)$ is $A{\mathbf{e}}_2$ where ${\mathbf{e}}_2=(0,1)$. The matrix encodes the transformation by recording where each basis vector goes.

Matrix operations

Matrix addition. Add entries componentwise (only for matrices of the same size):

$A+B=\left(\begin{array}{cc}{a}_{11}+b_{11}& a_{12}+b_{12}\\ a_{21}+b_{21}& a_{22}+b_{22}\end{array}\right)$

Scalar multiplication. Multiply every entry by the scalar:

$cA=\left(\begin{array}{cc}c{a}_{11}& c{a}_{12}\\ c{a}_{21}& c{a}_{22}\end{array}\right)$

Matrix multiplication. If $A$ is $m\times n$ and $B$ is $n\times p$, then $AB$ is $m\times p$ with entries:

$(AB{)}_{ij}={\sum }_{k=1}^n{a}_{ik}{b}_{kj}$

In words: the $(i,j)$ entry of $AB$ is the dot product of row $i$ of $A$ and column $j$ of $B$.

Example 3:

$A=\left(\begin{array}{cc}1& 2\\ 3& 4\end{array}\right),B=\left(\begin{array}{cc}5& 6\\ 7& 8\end{array}\right)$

$AB=\left(\begin{array}{cc}1\cdot 5+2\cdot 7& 1\cdot 6+2\cdot 8\\ 3\cdot 5+4\cdot 7& 3\cdot 6+4\cdot 8\end{array}\right)=\left(\begin{array}{cc}19& 22\\ 43& 50\end{array}\right)$

Why this strange definition? Because matrix multiplication corresponds to composition of linear transformations. If $A$ represents transformation $T_A$ and $B$ represents $T_B$, then $AB$ represents $T_A\circ T_B$ (apply $T_B$ first, then $T_A$).

Matrix multiplication is not commutative. In general, $AB\ne BA$. This is fundamental and follows from the fact that order matters in composition of transformations.

Identity matrix. The matrix $I$ with 1s on the diagonal and 0s elsewhere acts as the identity: $AI=IA=A$.

$I_3=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right)$

Matrix inverse. For a square matrix $A$, the inverse $A^{-1}$ is the matrix satisfying $A{A}^{-1}=A^{-1}A=I$. Not every matrix has an inverse. The matrix is invertible (or non-singular) if and only if its determinant is nonzero.

Transpose. The transpose $A^T$ swaps rows and columns: $(A^T{)}_{ij}=A_{ji}$.

Determinants

The determinant of a square matrix is a scalar that captures essential information about the transformation it represents.

For a 2×2 matrix:

$\det \left(\begin{array}{cc}a& b\\ c& d\end{array}\right)=ad-bc$

For a 3×3 matrix:

$\det \left(\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right)=a(ei-fh)-b(di-fg)+c(dh-eg)$

For larger matrices, expand by cofactors recursively. (In practice, computational software uses more efficient methods.)

What does the determinant measure?

For a 2×2 matrix: the signed area of the parallelogram spanned by the column vectors. For a 3×3 matrix: the signed volume of the parallelepiped. In $n$ dimensions: the signed $n$-volume.

The sign indicates orientation: positive means the transformation preserves orientation; negative means it reverses orientation (like a reflection).

Key facts:

• $\det A=0$ if and only if $A$ is not invertible. (Geometrically: the transformation collapses the volume to zero.)
• $\det (AB)=\det (A)\det (B)$ — determinants multiply.
• $\det (A^T)=\det (A)$ — determinants are preserved by transpose.

Linear independence, span, basis

Linear combination. A linear combination of vectors ${\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_k$ is any vector of the form

$c_1{\mathbf{v}}_1+c_2{\mathbf{v}}_2+\cdots +c_k{\mathbf{v}}_k$

for scalars $c_1,\dots ,c_k$.

Span. The span of a set of vectors is the set of all linear combinations of them. Geometrically: the line, plane, or higher-dimensional subspace they "fill out."

Linear independence. A set of vectors is linearly independent if no vector in the set can be written as a linear combination of the others. Equivalently: the only way to make a linear combination equal to zero is to use all-zero coefficients.

Basis. A basis for a vector space $V$ is a linearly independent set that spans $V$. Every vector in $V$ can be written uniquely as a linear combination of basis vectors.

Dimension. The dimension of a vector space is the number of vectors in any basis. (It does not depend on which basis you choose.) The standard $n$-dimensional Euclidean space has dimension $n$.

Example 4: In ${\mathbb{R}}^2$, the vectors $(1,0)$ and $(0,1)$ are linearly independent and span ${\mathbb{R}}^2$. So they form a basis. Any vector $(a,b)$ can be written as $a(1,0)+b(0,1)$.

Another basis for ${\mathbb{R}}^2$: $(1,1)$ and $(1,-1)$. These also span ${\mathbb{R}}^2$ and are linearly independent. The vector $(3,1)$ can be written as $2(1,1)+1(1,-1)$.

Vector spaces (briefly)

A vector space is a set $V$ equipped with two operations (vector addition and scalar multiplication) satisfying eight axioms:

1. Closure under addition: $\mathbf{u}+\mathbf{v}\in V$.
2. Closure under scalar multiplication: $c\mathbf{u}\in V$.
3. Commutativity of addition: $\mathbf{u}+\mathbf{v}=\mathbf{v}+\mathbf{u}$.
4. Associativity of addition: $(\mathbf{u}+\mathbf{v})+\mathbf{w}=\mathbf{u}+(\mathbf{v}+\mathbf{w})$.
5. Zero vector: $\exists 0$ such that $\mathbf{v}+0=\mathbf{v}$.
6. Additive inverses: for every $\mathbf{v}$, there exists $-\mathbf{v}$ such that $\mathbf{v}+(-\mathbf{v})=0$.
7. Distributivity: $c(\mathbf{u}+\mathbf{v})=c\mathbf{u}+c\mathbf{v}$, $(c+d)\mathbf{u}=c\mathbf{u}+d\mathbf{u}$.
8. Identity: $1\mathbf{v}=\mathbf{v}$.

These axioms abstract the properties of arrows-in-space. Anything satisfying them is a vector space. Examples include ${\mathbb{R}}^n$, the space of polynomials of degree $\le n$, the space of continuous functions, the space of $m\times n$ matrices.

In a serious mathematics course, you would work with these abstractions. For applied purposes, ${\mathbb{R}}^n$ is what you usually need.

Solving linear systems

The system $A\mathbf{x}=\mathbf{b}$ has different cases:

• Unique solution: $A$ is square and invertible. Solution is $\mathbf{x}=A^{-1}\mathbf{b}$.
• No solution: $\mathbf{b}$ is not in the column space of $A$. The system is inconsistent.
• Infinitely many solutions: $A$ has a non-trivial nullspace (vectors that map to zero). Any solution plus any null-space vector is also a solution.

Gaussian elimination is the standard algorithm: by row operations, transform $A$ to upper triangular form (called row echelon form), then solve by back-substitution.

Example 5: Solve

$\left(\begin{array}{cc}1& 1\\ 2& -1\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}5\\ 1\end{array}\right)$

Row 2 minus 2 times Row 1:

$\left(\begin{array}{cc}1& 1\\ 0& -3\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}5\\ -9\end{array}\right)$

From row 2: $-3y=-9$, so $y=3$. Back-substitute into row 1: $x+3=5$, so $x=2$. Solution: $(2,3)$.

Applications

Computer graphics. Every 3D transformation (rotation, scaling, translation) is implemented as a matrix multiplication. The graphics pipeline applies a sequence of matrices to vertices to produce the rendered image. GPUs are essentially matrix-multiplication accelerators.

Machine learning. Every layer of a neural network is a matrix multiplication followed by a nonlinear function. The trained "weights" of a network are entries of these matrices. Operations on data (PCA, embeddings, attention mechanisms) are matrix operations.

Physics. Quantum states are vectors in complex vector spaces (Hilbert spaces). Observables are matrices (operators). Quantum mechanics is essentially linear algebra applied to complex vector spaces.

Economics. Input-output analysis (Leontief models) uses matrices to track how production in one sector depends on inputs from others. Equilibria are computed by matrix inversion.

Statistics. The covariance matrix encodes the relationships between variables. Principal component analysis (PCA) finds eigenvectors of the covariance matrix.