n-grams to neural LMs
A language model assigns a probability to the next word. The oldest way to get one is to count.
Counting what comes next
A bigram model looks at the previous word only. From a corpus, count how often each word follows it and divide by the total: that is the probability of each next word. Generate text by sampling a word, then using it as the new context.
Pick a word in the lab to see its next-word distribution, then generate a sentence.
Why it isn’t enough
A bigram never sees further back than one word, so it loses the thread after a few words. Longer n-grams help a little, but most long contexts never appear in any corpus. Neural language models solve this by learning a function from the whole context to the next-token distribution, so similar contexts share what they learned.
A worked example
In the lab’s corpus “the” is followed six times: by “cat” twice and by “mat”, “dog”, “rug” and “fish” once each. A bigram model therefore says P(cat | the) = 2/6 ≈ 0.33 and P(dog | the) = 1/6 ≈ 0.17. A sentence’s probability is the product of such numbers, one per word.
Why counting runs out
With a vocabulary of 50,000 words there are 2.5 billion possible pairs and far more longer sequences, so most of them never appear in any corpus and get probability zero. Classic n-gram models patch this with smoothing, giving unseen sequences a little probability. Neural language models instead map words to embeddings, so a sentence about a “dog” informs predictions about a “cat”; transformers then extend the window from one or two words to thousands of tokens.
Easy to confuse
- An n-gram looks at the previous n − 1 words: a bigram uses one, a trigram two.
- Training on next-word prediction is the same task in a GPT-style model; what changed is how the prediction is computed, not what is predicted.
A bigram model predicts the next word from…
An n-gram model conditions on the last n − 1 words; for bigrams that is one word.
Why can’t we just use a 20-gram model for long context?
The number of possible contexts explodes; data never covers them, so counts are mostly empty. Neural models generalise across similar contexts instead.
What do neural language models keep from n-grams?
GPT-style models are still next-token predictors; they replace counting with a learned function of the whole context.
Read more in the books
- Speech and Language Processing (3rd ed. draft), Jurafsky, Martin: Ch. 3, “N-gram Language Models”.
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 12, “Applications”, section 12.4 (natural language processing, n-grams).
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 9, “Recurrent Neural Networks”, language models.