LLM Lab

Tokenisation

Models don’t read letters or words. They read tokens: chunks of text learned from data, usually pieces of words.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

Why subwords

Single characters make sequences long. Whole words make the vocabulary enormous and leave no way to write a word the model has never seen. Subword tokens sit in between: common words become one token, rare words are spelled from a few pieces.

Byte-pair encoding

BPE builds the vocabulary bottom-up. Start with single characters. Count every pair of neighbouring tokens across the corpus, merge the most frequent pair into a new token, and repeat until the vocabulary is the size you want.

n e w e s t _start: single characters
n e w es t _merge e + s (9 times in the corpus)
n e w est _merge es + t
n e w est_merge est + _
Byte-pair encoding on “newest”: each merge joins the most frequent neighbouring pair across the corpus, so common endings like “est” become single tokens.

Words in, IDs out

A tokenizer turns text into a list of integer IDs, one per token, and the model only ever sees those IDs. Token counts run higher than word counts: in the lecture’s example, 110 words became 162 tokens. Type in the first panel of the lab to see a toy tokenizer split your text.

Token3404izer7509s82·split6626·un555believ6667ably1346·long890·word1573s82.13
A sentence of 6 words becomes 11 tokens, each with an integer ID (the IDs here are illustrative). The model only ever sees the list of IDs.

The second panel runs BPE on a tiny corpus. Press Merge and watch frequent endings like est and whole words like low appear. The _ marks the end of a word.

A worked example

In the lab’s corpus, the pair e + s appears 9 times (6 in “newest”, 3 in “widest”), more than any other, so BPE merges it first. Then “es” + “t” becomes “est”, and so on. After enough merges, frequent words such as “low” are a single token while rarer ones are spelled from a few pieces.

In a language model

Real tokenizers learn tens of thousands of merges: GPT-2 has a vocabulary of about 50,000 tokens and LLaMA about 32,000. Most work on bytes, so any string, in any language, can be written with them. In English a token is on average about three quarters of a word; text in other languages, numbers and code usually take more tokens per word. Context windows, prices and speed are all counted in tokens, not words.

Easy to confuse

Check yourself · 1 of 3

Which pair does BPE merge next?

Check yourself · 2 of 3

Why not give every whole word its own token?

Check yourself · 3 of 3

A word the tokenizer has never seen…

Read more in the books

  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 2, “Working with Text Data”, section 2.5 (byte pair encoding).
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 2, tokens and token embeddings.
  • Speech and Language Processing (3rd ed. draft), Jurafsky, Martin: the chapter on tokenization.
All books
LabBPE merges
How to read this
  • Text box: type anything. Each coloured chip below it is one token (· marks a space), with its ID underneath: the number the model actually receives.
  • The line under the chips compares the number of words with the number of tokens.
  • BPE panel: each corpus word is shown split into its current tokens, with how often it occurs. “Most frequent pairs” counts neighbouring token pairs; Merge joins the top pair everywhere and adds it to the vocabulary.
corpus: low ×5 · lower ×2 · newest ×6 · widest ×3
Most frequent pairs