LLM Lab

Prompting & context

Everything the model knows about your task has to fit in its context window: instructions, examples, documents, history, and the answer it is about to write.

Where this sits in an LLM request
Instructions + context→Retrieved documents→Model generates→Output parsed and checked

The context is a budget

A model reads a fixed number of tokens at once. A prompt usually has several parts: a system prompt with standing instructions, examples of the desired output, retrieved documents, the conversation history, and room for the answer. In English, a token averages roughly four characters.

Spending it well

Clear, specific instructions do more than long ones. Examples are powerful but expensive. Documents and history grow fastest, so they are what you trim, summarise or retrieve more selectively. Adjust the parts in the lab and watch for overflow.

A worked example

An 8,192-token window: an 800-token system prompt, three examples of 600 tokens, three retrieved documents of 1,500, three earlier turns of 400 and 1,000 tokens kept for the answer add up to 9,300 tokens, which does not fit. Dropping one document and one old turn brings it to 7,400 and leaves room for the reply.

system 8003 examples 1,8003 documents 4,500history 1,200answer 1,000
An 8,192-token window spent on one request: 9,300 tokens are asked for here, so something must give, usually old history or the number of retrieved documents. Space for the answer has to be reserved too.

In practice

Put stable instructions first and the question last, separate parts clearly (headings or tags), and show the format you want with an example. More context is not always better: irrelevant documents distract the model and cost money on every call, and models tend to use the beginning and end of a long context better than the middle.

Easy to confuse

Check yourself · 1 of 3

The prompt fills the whole context window. What happens to the answer?

Check yourself · 2 of 3

A long chat starts forgetting early instructions. A common fix is…

Check yourself · 3 of 3

Few-shot examples help most when…

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 5, “Prompt Engineering”.
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 6, prompt engineering.
  • AI Engineering, Chip Huyen: Ch. 2, the section on context length and context efficiency.
All books
Labtoken budget bar
How to read this
  • Pick a context window (8K, 32K or 200K tokens). The bar shows how the window is filled by each part of the request.
  • Change the counts of examples, documents and conversation turns and the space reserved for the answer; the total line says how much room is left, or how far over the limit the request is.
Context window