Sampling
A model doesn’t output a word. It outputs a score for every token, and a sampler turns those scores into a choice.
Scores to probabilities
The last layer produces one score (a logit) per token in the vocabulary. Softmax turns them into probabilities that sum to 1: each token gets exp(logit / T), divided by the total.
The temperature T reshapes the distribution. Below 1 it sharpens it toward the top token; above 1 it flattens it, giving unlikely tokens more chance.
Cutting the tail
Top-k keeps only the k most likely tokens. Top-p keeps the smallest set of top tokens whose probabilities add up to p. Both renormalise what is left, so rare, odd tokens can’t be drawn at all.
Try it in the lab: raise the temperature and sample a few times, then switch on top-p to see the tail disappear.
A worked example
With the lab’s logits (3.2, 2.6, 2.2, 1.4, 0.9, 0.3) and T = 1, softmax gives mat 44.7%, floor 24.5%, sofa 16.4%, roof 7.4%, keyboard 4.5% and moon 2.5%. Top-k = 3 keeps mat, floor and sofa. Top-p = 0.9 adds tokens from the top until the total reaches 90%: mat, floor and sofa make 85.6%, so roof is added too (93.0%), and keyboard and moon are cut. The kept tokens are then rescaled to sum to 100%.
In practice
Greedy decoding (always the top token) suits tasks with one right answer, such as extraction, but tends to repeat itself in long text. Chat systems usually sample with a moderate temperature and top-p, which keeps answers varied without picking from the long, odd tail. The same model can feel very different under different settings, so evaluations should report them.
Easy to confuse
- Temperature changes how spread out the distribution is, never the order of the tokens.
- Top-k keeps a fixed number of tokens; top-p keeps a varying number, few when the model is confident and many when it is not.
Temperature 0.1 with top-k off. What do you expect when you sample?
Dividing logits by a small temperature stretches the gaps between them, so softmax puts nearly all the probability on the top token.
Top-p = 0.5 keeps which tokens?
Top-p (nucleus) sampling sorts tokens by probability and keeps adding them until the total reaches p, then renormalises.
Top-k = 1 is the same as…
Keeping only the single most likely token always picks the argmax, which is greedy decoding.
Read more in the books
- Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 5, “Pretraining on Unlabeled Data”, decoding strategies to control randomness.
- AI Engineering, Chip Huyen: Ch. 2, the section on sampling.
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 6, prompt engineering (generation parameters).