Tokenisation
Models don’t read letters or words. They read tokens: chunks of text learned from data, usually pieces of words.
Why subwords
Single characters make sequences long. Whole words make the vocabulary enormous and leave no way to write a word the model has never seen. Subword tokens sit in between: common words become one token, rare words are spelled from a few pieces.
Byte-pair encoding
BPE builds the vocabulary bottom-up. Start with single characters. Count every pair of neighbouring tokens across the corpus, merge the most frequent pair into a new token, and repeat until the vocabulary is the size you want.
Words in, IDs out
A tokenizer turns text into a list of integer IDs, one per token, and the model only ever sees those IDs. Token counts run higher than word counts: in the lecture’s example, 110 words became 162 tokens. Type in the first panel of the lab to see a toy tokenizer split your text.
The second panel runs BPE on a tiny corpus. Press Merge and watch frequent endings like est and whole words like low appear. The _ marks the end of a word.
A worked example
In the lab’s corpus, the pair e + s appears 9 times (6 in “newest”, 3 in “widest”), more than any other, so BPE merges it first. Then “es” + “t” becomes “est”, and so on. After enough merges, frequent words such as “low” are a single token while rarer ones are spelled from a few pieces.
In a language model
Real tokenizers learn tens of thousands of merges: GPT-2 has a vocabulary of about 50,000 tokens and LLaMA about 32,000. Most work on bytes, so any string, in any language, can be written with them. In English a token is on average about three quarters of a word; text in other languages, numbers and code usually take more tokens per word. Context windows, prices and speed are all counted in tokens, not words.
Easy to confuse
- A token ID is only a label. Its meaning comes from the embedding the model looks up for it, not from the number itself.
- The same word can split differently with a leading space, a capital letter or a typo, because those are different strings to the tokenizer.
Which pair does BPE merge next?
Byte-pair encoding greedily merges the most frequent adjacent pair, adds it to the vocabulary, and repeats.
Why not give every whole word its own token?
Subword tokens keep the vocabulary to tens of thousands while still spelling out any new or rare word from pieces.
A word the tokenizer has never seen…
Because the base vocabulary contains every character (or byte), any string can be encoded.
Read more in the books
- Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 2, “Working with Text Data”, section 2.5 (byte pair encoding).
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 2, tokens and token embeddings.
- Speech and Language Processing (3rd ed. draft), Jurafsky, Martin: the chapter on tokenization.