LLM Lab

Matrices & matrix multiplication

A matrix is a table of numbers. Multiplying two matrices is just a lot of dot products done together.

Where this maths shows up in a transformer
Embeddings (vectors)→Q·Kᵀ scores (matrices)→Softmax (probability)→Cross-entropy loss (logs)→Backprop (derivatives)→LayerNorm (mean, variance)

Matrices and shapes

A matrix with r rows and c columns has shape r × c. A linear layer is a matrix: it turns an input vector of size c into an output vector of size r. A tensor is the same idea with any number of dimensions. Keeping track of shapes is half of reading any model diagram.

Matrix multiplication

To multiply A (n × d) by B (d × m), the inner size d must match. Entry (i, j) of the result is the dot product of row i of A with column j of B, so the result is n × m. In attention, Q (N × d) times Kᵀ (d × N) gives the N × N table of scores in one step, which is why GPUs, built for matrix products, run transformers so well.

Press play in the lab to fill the result one cell at a time, or click any cell.

A worked example

Multiply [[1, 2], [3, 4]] by [[5, 6], [7, 8]]. The top-left entry is row 1 times column 1: 1·5 + 2·7 = 19. The top-right is row 1 times column 2: 1·6 + 2·8 = 22. Doing the same for row 2 gives 43 and 50, so the product is [[19, 22], [43, 50]]. Swap the order and you get [[23, 34], [31, 46]]: matrix multiplication does not commute.

1234
×
5678
=
19224350
The outlined row of the first matrix and the outlined column of the second give the outlined entry: 1·5 + 2·7 = 19. Each of the four entries is one such dot product.

A layer is a matrix

Multiplying a matrix W (out × in) by a vector x (size in) gives a vector of size out: each output number is one row of W dotted with x. That is a linear layer. Its weights are the entries of W, and training adjusts them. In a 7-billion-parameter LLaMA, the query projection of one layer is a 4096 × 4096 matrix and the feed-forward layers are 4096 × 11008 and back.

·
=
A linear layer: W (3 × 4) times an input vector of 4 numbers gives an output of 3 numbers. Each output is one row of W dotted with the input.

Why shapes and cost matter

The transpose Kᵀ swaps rows and columns, turning N × d into d × N, which is why attention writes Q·Kᵀ: the inner sizes then match. Multiplying n × d by d × m takes n·d·m multiply-adds. For attention scores that is N·d·N: doubling the context length quadruples the work, one reason long contexts are expensive.

Easy to confuse

Check yourself · 1 of 3

A is 4 × 3 and B is 3 × 5. What is the shape of A·B?

Check yourself · 2 of 3

How is one entry of A·B computed?

Check yourself · 3 of 3

Q is N × d and K is N × d. What shape is Q·Kᵀ?

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 2, “Linear Algebra”: multiplying matrices and vectors.
  • Mathematics for Machine Learning, Deisenroth, Faisal, Ong: Ch. 2, “Linear Algebra”.
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 22, “Appendix: Mathematics for Deep Learning”: geometry and linear algebraic operations.
All books
Labpick an output cell
How to read this
  • Three grids: A (4 × 3), B (3 × 2) and the result A·B (4 × 2). Every cell is one number; teal is positive, orange negative.
  • The outlined row of A and the outlined column of B are the pair being combined. Their dot product lands in the outlined cell of A·B.
  • Hatched cells of A·B are not computed yet. Press Play to fill them in order, or click any cell to jump to it.
  • The box spells out the row × column sum for the selected cell.