LLM Lab

Vectors & dot products

A vector is a list of numbers. Almost everything in a language model is one, and the dot product tells you how much two of them agree.

Where this maths shows up in a transformer
Embeddings (vectors)→Q·Kᵀ scores (matrices)→Softmax (probability)→Cross-entropy loss (logs)→Backprop (derivatives)→LayerNorm (mean, variance)

Vectors

A vector is an ordered list of numbers, such as (3, 1). You can picture it as an arrow from the origin, or as a strip of cells, one per number. A word embedding is a vector with hundreds or thousands of numbers instead of two; the lab draws positive numbers in teal and negative ones in orange.

The dot product

Multiply the vectors number by number, then add: a · b = a₁b₁ + a₂b₂ + … Geometrically it equals |a| |b| cos θ, where θ is the angle between them: large when they point the same way, zero when perpendicular, negative when they point apart. Divide by both lengths and you get the cosine similarity, which ignores length.

This one operation is behind attention: every attention score in a transformer is the dot product of a query with a key.

A worked example

Take a = (2, 1) and b = (1, 3). The dot product is 2·1 + 1·3 = 5. The lengths are |a| = √(4 + 1) ≈ 2.24 and |b| = √(1 + 9) ≈ 3.16, so cos θ = 5 / (2.24 × 3.16) ≈ 0.71 and the angle is about 45°. The two vectors point in fairly similar directions.

a = (2, 1)b = (1, 3)
a21
b13
× each23
sum5
cos θ5 ÷ (2.24 × 3.16) ≈ 0.71
a · b: multiply the two vectors number by number (2·1 and 1·3), then add (2 + 3 = 5). Dividing by both lengths gives cos θ ≈ 0.71, an angle of about 45°.

Geometrically, a · b is the length of a times the length of b’s shadow on a. That is why the sign tells you the direction: positive when b leans the same way as a, zero when it is perpendicular, negative when it leans away. Length matters too: double b and the dot product doubles, which is why cosine similarity divides both lengths out.

In a language model

Embeddings are long vectors: 768 numbers per token in GPT-2 small, 4096 in a 7-billion-parameter LLaMA. Every attention score is a dot product of a query with a key, computed over the numbers of one head (128 per head in Llama 2). Retrieval systems rank documents by the cosine similarity of their embeddings with the question’s embedding.

One more detail explains a formula you will meet later. The dot product of two random d-number vectors tends to grow like √d, so attention divides its scores by √d to keep them in a stable range whatever the vector size.

Easy to confuse

Check yourself · 1 of 3

Two vectors point in the same direction. What is their cosine similarity?

Check yourself · 2 of 3

a = (1, 0) and b = (0, 3). What is a · b?

Check yourself · 3 of 3

Why do attention and embedding search use the dot product?

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 2, “Linear Algebra”.
  • Mathematics for Machine Learning, Deisenroth, Faisal, Ong: Ch. 2, “Linear Algebra” and Ch. 3, “Analytic Geometry”.
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 2, “Preliminaries”: linear algebra.
All books
Labangle and similarity
How to read this
  • Top: two arrows in a plane. a stays fixed; the slider turns b. The dashed arc marks the angle θ between them.
  • The box under the plane writes out a · b = a₁b₁ + a₂b₂ with the current numbers, then cos θ.
  • Bottom: each word is a 6-number vector drawn as a strip of cells. Teal cells are positive, orange negative; a darker cell is a larger number. Click two words to compare them.
  • The “× each” strip multiplies the two words number by number. Adding that strip gives the dot product; dividing by both lengths gives the cosine.
In 2 dimensions: turn vector b
In 6 dimensions: compare two word vectors