Inference Performance: The Metrics That Actually Matter
Tokens per second is the headline number, but production inference quality is decided by a family of metrics — and by whether they are measured under realistic concurrency. A practical guide to what to measure and why.
The four numbers that define user experience
For streaming model responses, users experience four quantities: time-to-first-token (how quickly the response begins), inter-token latency (how smoothly it streams), total completion time, and availability under concurrency. Optimizing any one in isolation distorts the others — aggressive batching raises throughput but inflates time-to-first-token; aggressive quantization raises speed but can degrade quality. Performance engineering is the discipline of holding all four within targets simultaneously.
The targets themselves should come from the product, not from infrastructure folklore. A retrieval-grounded assistant may tolerate higher time-to-first-token because users expect retrieval; an interactive coding assistant may not.
Benchmark under realistic conditions
The most common benchmarking failure is measuring under conditions that never occur in production: single requests, synthetic prompts, no cache pressure. Meaningful inference benchmarks reproduce realistic concurrency, realistic prompt and output length distributions, and the mixture of workloads the endpoint actually serves.
Equally important is benchmark stability. Benchmarks run only before launch become folklore; benchmarks that run continuously — against the same suite, on every model and configuration change — become the system of record that makes optimization safe.
The bottleneck is rarely the GPU
Teams optimizing inference often reach for the GPU first. In practice the constraint is frequently elsewhere: tokenization and preprocessing on CPU, retrieval calls adding fixed latency to every request, network hops between services, or serialization overhead in the application layer. A profile that attributes end-to-end latency to each layer usually redirects effort immediately.
KV-cache management deserves special attention for LLM serving: cache sizing, eviction policy and prefix reuse determine how much concurrency a GPU can sustain. Serving stack configuration — continuous batching, scheduling policy, memory allocation — routinely moves throughput more than hardware selection does.
Cost per unit of work
Performance without cost context is half a story. Cost per request, per thousand tokens, or per completed task turns performance into economics, and makes trade-offs discussable: quantization choices, model selection, caching strategies and infrastructure placement all become measurable decisions rather than preferences.
The most useful framing is a unit-economics dashboard per workload: latency distribution, throughput, quality metrics and cost per unit, tracked over time. When a model version changes, all four move together — and the team can see whether the change improved the system or merely moved cost around.
Building the practice
A durable inference performance practice has three parts: a benchmark suite that reflects real traffic, continuous measurement wired into deployment gates, and a unit-economics view shared by engineering and finance. None requires exotic tooling; all require consistency.
Organizations that build this practice make faster, calmer optimization decisions — because the numbers are already there when the debate starts.
- AI Infrastructure
- Enterprise AI
- Private AI
- Inference Performance
- Hybrid Cloud