Prompting & context
Everything the model knows about your task has to fit in its context window: instructions, examples, documents, history, and the answer it is about to write.
The context is a budget
A model reads a fixed number of tokens at once. A prompt usually has several parts: a system prompt with standing instructions, examples of the desired output, retrieved documents, the conversation history, and room for the answer. In English, a token averages roughly four characters.
Spending it well
Clear, specific instructions do more than long ones. Examples are powerful but expensive. Documents and history grow fastest, so they are what you trim, summarise or retrieve more selectively. Adjust the parts in the lab and watch for overflow.
A worked example
An 8,192-token window: an 800-token system prompt, three examples of 600 tokens, three retrieved documents of 1,500, three earlier turns of 400 and 1,000 tokens kept for the answer add up to 9,300 tokens, which does not fit. Dropping one document and one old turn brings it to 7,400 and leaves room for the reply.
In practice
Put stable instructions first and the question last, separate parts clearly (headings or tags), and show the format you want with an example. More context is not always better: irrelevant documents distract the model and cost money on every call, and models tend to use the beginning and end of a long context better than the middle.
Easy to confuse
- The answer counts against the same window as the prompt.
- A longer prompt is not a better prompt; a clear task, constraints and an example usually matter more than length.
The prompt fills the whole context window. What happens to the answer?
Input and output tokens count against one limit, so you must reserve room for the response.
A long chat starts forgetting early instructions. A common fix is…
History grows every turn; trimming or summarising it keeps the important parts in view.
Few-shot examples help most when…
Examples are a strong signal of format and style, but each one costs tokens.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 5, “Prompt Engineering”.
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 6, prompt engineering.
- AI Engineering, Chip Huyen: Ch. 2, the section on context length and context efficiency.