LLM Lab

Embeddings

Before a model can compute with a token, it turns it into a vector. Tokens used in similar ways end up close together.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

A table of vectors

The simplest way to turn a word into numbers is one-hot encoding: a vector as long as the vocabulary, all zeros except a 1 at the word’s index. It is long, sparse, and says nothing about which words are alike. A word embedding instead maps the word’s index to a short dense vector through a lookup table, one learned row per token. Multiplying a one-hot vector by the table picks out exactly that row.

one-hot
table
thecatking…
result2.23.40.4
One-hot: 12 numbers (one per word in the vocabulary), all zero but one. Multiplying it by the embedding table picks out that word’s row: a short, dense vector.

The table is trained with the rest of the model, so tokens that appear in similar contexts end up with similar vectors. Older methods such as Word2Vec and GloVe trained such tables on their own.

Similarity and directions

Cosine similarity compares the directions of two vectors: 1 means the same direction, 0 unrelated. Some relations become directions too, so king − man + woman lands near queen.

king = (2.2, 3.4)man = (1.8, 1.6)woman = (3.2, 1.5)queen = (3.6, 3.3)
In the lab’s toy map, king − man + woman = (2.2 − 1.8 + 3.2, 3.4 − 1.6 + 1.5) = (3.6, 3.3), exactly where “queen” sits. Real embeddings land near, not on, the answer.

The lab uses two dimensions so you can see them; the coordinates are hand-made for illustration. Click a word to rank its neighbours, or run the analogy.

In a language model

Real embeddings have hundreds or thousands of numbers per token: 768 in GPT-2 small, 4096 in a 7-billion-parameter LLaMA. The table alone is large: 32,000 tokens × 4,096 numbers is about 131 million parameters. The vectors are learned during pretraining together with everything else, and some models reuse the same table for the final layer that turns vectors back into token scores.

Easy to confuse

Check yourself · 1 of 3

What does it mean for two embeddings to have high cosine similarity?

Check yourself · 2 of 3

king − man + woman ≈ queen works because…

Check yourself · 3 of 3

Where do embeddings come from in an LLM?

Read more in the books

  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 15, “Natural Language Processing: Pretraining”, word embedding (word2vec).
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 2, tokens and token embeddings.
  • Speech and Language Processing (3rd ed. draft), Jurafsky, Martin: the chapter on embeddings.
All books
Labvector map
How to read this
  • The map places each word at its two numbers. Click a word to select it; the readout ranks its nearest words by cosine similarity.
  • The analogy buttons draw the vector arithmetic as an arrow and name the word closest to the result.
  • Below the map: the selected word as a one-hot vector (one lit cell out of 12), then the embedding table with one row per word. The outlined row is what the lookup returns.