Embeddings
Before a model can compute with a token, it turns it into a vector. Tokens used in similar ways end up close together.
A table of vectors
The simplest way to turn a word into numbers is one-hot encoding: a vector as long as the vocabulary, all zeros except a 1 at the word’s index. It is long, sparse, and says nothing about which words are alike. A word embedding instead maps the word’s index to a short dense vector through a lookup table, one learned row per token. Multiplying a one-hot vector by the table picks out exactly that row.
The table is trained with the rest of the model, so tokens that appear in similar contexts end up with similar vectors. Older methods such as Word2Vec and GloVe trained such tables on their own.
Similarity and directions
Cosine similarity compares the directions of two vectors: 1 means the same direction, 0 unrelated. Some relations become directions too, so king − man + woman lands near queen.
The lab uses two dimensions so you can see them; the coordinates are hand-made for illustration. Click a word to rank its neighbours, or run the analogy.
In a language model
Real embeddings have hundreds or thousands of numbers per token: 768 in GPT-2 small, 4096 in a 7-billion-parameter LLaMA. The table alone is large: 32,000 tokens × 4,096 numbers is about 131 million parameters. The vectors are learned during pretraining together with everything else, and some models reuse the same table for the final layer that turns vectors back into token scores.
Easy to confuse
- The two-dimensional map is a teaching device. Real embedding spaces have no meaningful “x” and “y”, and analogies land near the answer rather than exactly on it.
- An embedding of a token is fixed; what changes with context is the vector after the transformer blocks have mixed in the surrounding tokens.
What does it mean for two embeddings to have high cosine similarity?
Cosine similarity measures the angle between vectors, ignoring length; related meanings end up pointing the same way.
king − man + woman ≈ queen works because…
In good embedding spaces some relations become roughly constant offsets, so vector arithmetic can follow them.
Where do embeddings come from in an LLM?
The embedding matrix is a set of parameters updated by gradient descent like every other weight.
Read more in the books
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 15, “Natural Language Processing: Pretraining”, word embedding (word2vec).
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 2, tokens and token embeddings.
- Speech and Language Processing (3rd ed. draft), Jurafsky, Martin: the chapter on embeddings.