LLM Lab

n-grams to neural LMs

A language model assigns a probability to the next word. The oldest way to get one is to count.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

Counting what comes next

A bigram model looks at the previous word only. From a corpus, count how often each word follows it and divide by the total: that is the probability of each next word. Generate text by sampling a word, then using it as the new context.

cat2/6 = 33%
mat1/6 = 17%
dog1/6 = 17%
rug1/6 = 17%
fish1/6 = 17%
P(next word | “the”) in the lab’s corpus: “the” is followed six times, twice by “cat”. A bigram model simply turns these counts into probabilities.

Pick a word in the lab to see its next-word distribution, then generate a sentence.

Why it isn’t enough

A bigram never sees further back than one word, so it loses the thread after a few words. Longer n-grams help a little, but most long contexts never appear in any corpus. Neural language models solve this by learning a function from the whole context to the next-token distribution, so similar contexts share what they learned.

A worked example

In the lab’s corpus “the” is followed six times: by “cat” twice and by “mat”, “dog”, “rug” and “fish” once each. A bigram model therefore says P(cat | the) = 2/6 ≈ 0.33 and P(dog | the) = 1/6 ≈ 0.17. A sentence’s probability is the product of such numbers, one per word.

Why counting runs out

With a vocabulary of 50,000 words there are 2.5 billion possible pairs and far more longer sequences, so most of them never appear in any corpus and get probability zero. Classic n-gram models patch this with smoothing, giving unseen sequences a little probability. Neural language models instead map words to embeddings, so a sentence about a “dog” informs predictions about a “cat”; transformers then extend the window from one or two words to thousands of tokens.

Easy to confuse

Check yourself · 1 of 3

A bigram model predicts the next word from…

Check yourself · 2 of 3

Why can’t we just use a 20-gram model for long context?

Check yourself · 3 of 3

What do neural language models keep from n-grams?

Read more in the books

  • Speech and Language Processing (3rd ed. draft), Jurafsky, Martin: Ch. 3, “N-gram Language Models”.
  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 12, “Applications”, section 12.4 (natural language processing, n-grams).
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 9, “Recurrent Neural Networks”, language models.
All books
Labpredict the next word
How to read this
  • The line at the top is the whole training corpus.
  • Click a word to see the distribution of the word that follows it: each bar is a count divided by the total.
  • Generate samples a sentence one word at a time, each time drawing from the bars of the previous word.
corpus: the cat sat on the mat. the dog sat on the rug. the cat ate the fish. a dog ate a bone.
…