Matrices & matrix multiplication
A matrix is a table of numbers. Multiplying two matrices is just a lot of dot products done together.
Matrices and shapes
A matrix with r rows and c columns has shape r × c. A linear layer is a matrix: it turns an input vector of size c into an output vector of size r. A tensor is the same idea with any number of dimensions. Keeping track of shapes is half of reading any model diagram.
Matrix multiplication
To multiply A (n × d) by B (d × m), the inner size d must match. Entry (i, j) of the result is the dot product of row i of A with column j of B, so the result is n × m. In attention, Q (N × d) times Kᵀ (d × N) gives the N × N table of scores in one step, which is why GPUs, built for matrix products, run transformers so well.
Press play in the lab to fill the result one cell at a time, or click any cell.
A worked example
Multiply [[1, 2], [3, 4]] by [[5, 6], [7, 8]]. The top-left entry is row 1 times column 1: 1·5 + 2·7 = 19. The top-right is row 1 times column 2: 1·6 + 2·8 = 22. Doing the same for row 2 gives 43 and 50, so the product is [[19, 22], [43, 50]]. Swap the order and you get [[23, 34], [31, 46]]: matrix multiplication does not commute.
A layer is a matrix
Multiplying a matrix W (out × in) by a vector x (size in) gives a vector of size out: each output number is one row of W dotted with x. That is a linear layer. Its weights are the entries of W, and training adjusts them. In a 7-billion-parameter LLaMA, the query projection of one layer is a 4096 × 4096 matrix and the feed-forward layers are 4096 × 11008 and back.
Why shapes and cost matter
The transpose Kᵀ swaps rows and columns, turning N × d into d × N, which is why attention writes Q·Kᵀ: the inner sizes then match. Multiplying n × d by d × m takes n·d·m multiply-adds. For attention scores that is N·d·N: doubling the context length quadruples the work, one reason long contexts are expensive.
Easy to confuse
- The shape rule is about the inner sizes. A (4 × 3) · B (3 × 2) works; B · A (3 × 2 · 4 × 3) does not.
- A·B is not entry-by-entry multiplication. That operation, written A ⊙ B, needs equal shapes and appears in gated layers such as GLU.
A is 4 × 3 and B is 3 × 5. What is the shape of A·B?
The inner sizes (3 and 3) must match and disappear; the outer sizes remain: 4 × 5.
How is one entry of A·B computed?
Entry (i, j) of A·B is row i of A dotted with column j of B. A matrix product is many dot products at once.
Q is N × d and K is N × d. What shape is Q·Kᵀ?
Kᵀ is d × N, so Q·Kᵀ is N × N: one score for every pair of tokens. That is the attention score matrix.
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 2, “Linear Algebra”: multiplying matrices and vectors.
- Mathematics for Machine Learning, Deisenroth, Faisal, Ong: Ch. 2, “Linear Algebra”.
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 22, “Appendix: Mathematics for Deep Learning”: geometry and linear algebraic operations.