Latency, throughput & cost
A model can be accurate and still unusable if it is slow or expensive. Serving metrics measure speed and cost per request.
Latency: two phases
A request has two phases. In prefill the model reads the whole prompt in parallel and produces the first token; the wait for it is the time to first token (TTFT). In decode it generates one token per step; the gap between tokens is the time per output token (TPOT). Total latency ≈ TTFT + output tokens × TPOT.
Throughput and cost
Throughput is how many tokens or requests a server completes per second across all users; batching many requests raises throughput but can raise each user’s latency. Cost per request is input tokens × input price + output tokens × output price. Teams often track goodput: the requests per second that still meet a latency target.
A worked example
A 2,000-token prompt and a 300-token answer, with prefill at 5,000 tokens/s and decode at 60 tokens/s: TTFT ≈ 0.4 s, decoding takes 5 s, so the answer is complete after about 5.4 s. At example prices of $3 per million input tokens and $15 per million output tokens, the request costs 0.006 + 0.0045 ≈ $0.0105, about 95 requests per dollar. Long retrieved contexts raise both TTFT and cost, which is why trimming context is a serving optimisation too.
Easy to confuse
- Latency is per request; throughput is for the whole server. Improving one can worsen the other.
- Report percentiles (p50, p95), not only averages: a few slow requests are what users remember.
A chat model takes 0.4 s before the first token and then streams 50 tokens per second. How long until a 200-token answer is complete?
Total latency is time to first token plus output tokens divided by decode speed: 0.4 + 200 / 50 = 4.4 s.
Which number matters most for how fast a streamed chat answer feels?
With streaming, users start reading as soon as the first token appears, so TTFT dominates perceived speed; decode speed then needs only to stay ahead of reading.
Why are output tokens usually priced higher than input tokens?
The prompt is processed in one parallel pass (prefill), but each output token needs its own step through the model (decode), which uses the hardware less efficiently.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 9, “Inference Optimization”: inference performance metrics (latency, TTFT, TPOT, throughput).