LLM Lab

Sampling

A model doesn’t output a word. It outputs a score for every token, and a sampler turns those scores into a choice.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

Scores to probabilities

The last layer produces one score (a logit) per token in the vocabulary. Softmax turns them into probabilities that sum to 1: each token gets exp(logit / T), divided by the total.

The temperature T reshapes the distribution. Below 1 it sharpens it toward the top token; above 1 it flattens it, giving unlikely tokens more chance.

Cutting the tail

Top-k keeps only the k most likely tokens. Top-p keeps the smallest set of top tokens whose probabilities add up to p. Both renormalise what is left, so rare, odd tokens can’t be drawn at all.

Try it in the lab: raise the temperature and sample a few times, then switch on top-p to see the tail disappear.

A worked example

With the lab’s logits (3.2, 2.6, 2.2, 1.4, 0.9, 0.3) and T = 1, softmax gives mat 44.7%, floor 24.5%, sofa 16.4%, roof 7.4%, keyboard 4.5% and moon 2.5%. Top-k = 3 keeps mat, floor and sofa. Top-p = 0.9 adds tokens from the top until the total reaches 90%: mat, floor and sofa make 85.6%, so roof is added too (93.0%), and keyboard and moon are cut. The kept tokens are then rescaled to sum to 100%.

mat44.7% · cumulative 45%
floor24.5% · cumulative 69%
sofa16.4% · cumulative 86%
roof7.4% · cumulative 93%
keyboard4.5% · cumulative 98% · cut by top-p 0.9
moon2.5% · cumulative 100% · cut by top-p 0.9
Next-token probabilities for “The cat sat on the ___” at T = 1. Top-p = 0.9 keeps the smallest set of top tokens whose total reaches 90% (four here), then renormalises them.

In practice

Greedy decoding (always the top token) suits tasks with one right answer, such as extraction, but tends to repeat itself in long text. Chat systems usually sample with a moderate temperature and top-p, which keeps answers varied without picking from the long, odd tail. The same model can feel very different under different settings, so evaluations should report them.

Easy to confuse

Check yourself · 1 of 3

Temperature 0.1 with top-k off. What do you expect when you sample?

Check yourself · 2 of 3

Top-p = 0.5 keeps which tokens?

Check yourself · 3 of 3

Top-k = 1 is the same as…

Read more in the books

  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 5, “Pretraining on Unlabeled Data”, decoding strategies to control randomness.
  • AI Engineering, Chip Huyen: Ch. 2, the section on sampling.
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 6, prompt engineering (generation parameters).
All books
Labtemperature & top-p
How to read this
  • The stage map shows the path from the model’s last layer to the chosen token; the stage you last touched lights up.
  • Each bar is the probability of one candidate for “The cat sat on the ___”. Tokens removed by top-k or top-p drop to zero and the rest are renormalised.
  • Sample draws one token from the bars; the line next to it counts what you have drawn so far.
The cat sat on the ___
Top-k
Top-p
no samples yet