LLM Lab

Latency, throughput & cost

A model can be accurate and still unusable if it is slow or expensive. Serving metrics measure speed and cost per request.

Where this sits when evaluating a model
Choose test data→Score the outputs→Compare models→Check trust (calibration, leaks)→Evaluate whole systems→Measure speed and cost

Latency: two phases

A request has two phases. In prefill the model reads the whole prompt in parallel and produces the first token; the wait for it is the time to first token (TTFT). In decode it generates one token per step; the gap between tokens is the time per output token (TPOT). Total latency ≈ TTFT + output tokens × TPOT.

TTFTdecode: 200 tokens at 50 tokens/s = 4.0 s
One request on a timeline: the prompt is processed first (prefill, ending at the first token after 0.4 s here), then tokens stream out one by one. Total latency 4.4 s; time per output token 20 ms.

Throughput and cost

Throughput is how many tokens or requests a server completes per second across all users; batching many requests raises throughput but can raise each user’s latency. Cost per request is input tokens × input price + output tokens × output price. Teams often track goodput: the requests per second that still meet a latency target.

A worked example

A 2,000-token prompt and a 300-token answer, with prefill at 5,000 tokens/s and decode at 60 tokens/s: TTFT ≈ 0.4 s, decoding takes 5 s, so the answer is complete after about 5.4 s. At example prices of $3 per million input tokens and $15 per million output tokens, the request costs 0.006 + 0.0045 ≈ $0.0105, about 95 requests per dollar. Long retrieved contexts raise both TTFT and cost, which is why trimming context is a serving optimisation too.

Easy to confuse

Check yourself · 1 of 3

A chat model takes 0.4 s before the first token and then streams 50 tokens per second. How long until a 200-token answer is complete?

Check yourself · 2 of 3

Which number matters most for how fast a streamed chat answer feels?

Check yourself · 3 of 3

Why are output tokens usually priced higher than input tokens?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 9, “Inference Optimization”: inference performance metrics (latency, TTFT, TPOT, throughput).
All books
Labtime one request
How to read this
  • Set the prompt and answer lengths, how fast the server processes the prompt (prefill) and generates tokens (decode), and example prices per million tokens.
  • The bar is one request on a timeline: orange is the wait for the first token, teal the streaming of the answer. The tiles give time to first token, total latency, cost per request and requests per dollar.