Vectors & dot products
A vector is a list of numbers. Almost everything in a language model is one, and the dot product tells you how much two of them agree.
Vectors
A vector is an ordered list of numbers, such as (3, 1). You can picture it as an arrow from the origin, or as a strip of cells, one per number. A word embedding is a vector with hundreds or thousands of numbers instead of two; the lab draws positive numbers in teal and negative ones in orange.
The dot product
Multiply the vectors number by number, then add: a · b = a₁b₁ + a₂b₂ + … Geometrically it equals |a| |b| cos θ, where θ is the angle between them: large when they point the same way, zero when perpendicular, negative when they point apart. Divide by both lengths and you get the cosine similarity, which ignores length.
This one operation is behind attention: every attention score in a transformer is the dot product of a query with a key.
A worked example
Take a = (2, 1) and b = (1, 3). The dot product is 2·1 + 1·3 = 5. The lengths are |a| = √(4 + 1) ≈ 2.24 and |b| = √(1 + 9) ≈ 3.16, so cos θ = 5 / (2.24 × 3.16) ≈ 0.71 and the angle is about 45°. The two vectors point in fairly similar directions.
Geometrically, a · b is the length of a times the length of b’s shadow on a. That is why the sign tells you the direction: positive when b leans the same way as a, zero when it is perpendicular, negative when it leans away. Length matters too: double b and the dot product doubles, which is why cosine similarity divides both lengths out.
In a language model
Embeddings are long vectors: 768 numbers per token in GPT-2 small, 4096 in a 7-billion-parameter LLaMA. Every attention score is a dot product of a query with a key, computed over the numbers of one head (128 per head in Llama 2). Retrieval systems rank documents by the cosine similarity of their embeddings with the question’s embedding.
One more detail explains a formula you will meet later. The dot product of two random d-number vectors tends to grow like √d, so attention divides its scores by √d to keep them in a stable range whatever the vector size.
Easy to confuse
- The dot product is a single number, not a vector. The element-wise product (the “× each” strip) only becomes the dot product once you add it up.
- Cosine similarity ignores length; the dot product does not. Two vectors in the same direction can have a small or large dot product.
Two vectors point in the same direction. What is their cosine similarity?
Cosine similarity is the cosine of the angle between the vectors. Same direction means an angle of 0°, and cos 0° = 1, whatever the lengths are.
a = (1, 0) and b = (0, 3). What is a · b?
The dot product is 1·0 + 0·3 = 0. The vectors are perpendicular, so they share nothing in each other’s direction.
Why do attention and embedding search use the dot product?
A large positive dot product means two vectors point the same way, which the model reads as “relevant to each other”. Attention scores are dot products of queries and keys.
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 2, “Linear Algebra”.
- Mathematics for Machine Learning, Deisenroth, Faisal, Ong: Ch. 2, “Linear Algebra” and Ch. 3, “Analytic Geometry”.
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 2, “Preliminaries”: linear algebra.