Metrics, and How to Measure Them Without Lying to Yourself

Part 0 set an objective: maximise the number of tenants served within their SLA, per dollar, on fixed hardware. Every post after this one claims to move that number.

Which means this post has to come first, because a claim like “prefix caching bought us four more tenants” is worth exactly as much as the measurement behind it. Get the measurement wrong and the rest of the series is a sequence of confident assertions about nothing.

So: the scoreboard. What each number means, what it hides, and the specific ways a benchmark on one contended machine will mislead you if you let it.

The two latencies

“Latency” is not one number for a streaming model. A request has three phases and two distinct waits, and collapsing them loses the thing you care about.

The two latencies of one request

One request, three phases. A single average latency number describes none of them — and hides the queue entirely.

Time to first token (TTFT) is how long the user stares at nothing. It covers queueing, admission, and prefill — the whole path from arrival to the first visible character.

Inter-token latency (ITL) is the gap between tokens after that. It’s what the reader experiences as the speed the text appears, and it’s governed by decode: one token per step, each step reading every weight out of HBM.

They move independently, and often in opposite directions. Batching more requests together raises throughput and generally improves nobody’s TTFT; a larger batch makes each decode step do more work, so ITL widens even as the machine gets more efficient. A change that improves one and degrades the other is the normal case, not the exception, which is why a single latency number can’t govern any decision in this series.

Measure TTFT from arrival, not from prefill

The engine’s own timers usually start when a request begins executing. That excludes the queue — and on a box that is deliberately run near capacity, the queue is the story. A TTFT that looks flat while tenants complain is almost always a TTFT measured from the wrong instant.

Throughput, and the question it hides

Throughput comes in two units and neither means anything alone.

Tokens per second is the honest measure of what the chip is doing. Requests per second is what capacity planning wants. Both need a qualifier that is almost always omitted: at what concurrency, and within what latency bound?

Throughput without a latency bound is trivially gamed. Queue a thousand requests, batch them as wide as HBM allows, and tokens/sec goes up while every individual tenant waits far longer than their SLA permits. That configuration is “faster” by the headline number and useless by the objective in Part 0.

So throughput is only ever reported here as a pair:

N tokens/sec at C concurrent requests with p99 TTFT under T.

Drop any of the three and the number stops being a measurement and becomes a boast.

Percentiles, and why they have to be per tenant

Averages are useless here, for the ordinary reason and one specific to multi-tenancy.

The ordinary reason: latency distributions in a batching server are not remotely normal. A mean sits between the fast path and the slow path and describes neither.

The specific reason is the one this series exists for. Imagine ten tenants; nine are served well and one is being starved by an eviction policy that keeps dropping its blocks. Aggregate p99 covers all requests across all tenants — so if the starved tenant sends 5% of traffic, its catastrophic experience is a rounding error inside a healthy-looking global percentile. The number stays green while a tenant leaves.

The rule for the rest of the series

Every latency number is a percentile — p50, p95, p99 — and every percentile is computed per tenant, then reported as a distribution across tenants. “p99 TTFT” on its own is not a result. “Worst tenant’s p99 TTFT” is.

This costs nothing to implement and changes what you can see. It is the difference between a fairness claim and a fairness measurement.

HBM headroom: the one that only binds here

This is the metric most serving write-ups don’t carry, because on a cluster it never binds — when the pool gets tight you add a node and the question dissolves.

On one chip it is the leading indicator for everything else. Part 0 put the number at roughly 3.8 GiB of KV pool after weights. Track, continuously:

Free KV blocks as a fraction of the pool
Preemptions and evictions per minute — requests whose blocks were reclaimed mid-flight
Admission rejections — requests refused rather than queued

The relationship worth internalizing: headroom is a leading indicator and latency is a lagging one. By the time p99 TTFT moves, the pool has been thrashing for a while. A benchmark that only watches latency finds out last.

MFU, and why a low number is the expected answer

Model FLOPs utilization — how much of the chip’s arithmetic capacity you actually used — is the standard efficiency metric, and it needs recalibrating before it’s useful here.

In training, high MFU is the goal and 40–60% is respectable. In decode, MFU is structurally low and there is no configuration that fixes it, because decode is memory-bound: the chip spends its time reading weights out of HBM, not multiplying. Post 2’s section 13 works through why, and Part 3 will do the TPU-specific version.

So a decode MFU of a few percent is not a bug report. Used correctly, MFU here answers a narrower question: did this change move the chip closer to being compute-bound? Batching should raise it. Quantization should raise it. If a change that was supposed to improve batching leaves MFU flat, the batching didn’t happen — which is a far more useful signal than the absolute value ever is.

Measuring without lying to yourself

Four ways a benchmark on this machine will mislead you.

Warm up, or you’re benchmarking the compiler

XLA compiles per input shape, and the first request of any new shape pays for a compilation. Measure from a cold engine and you’ve measured compilation, then averaged it into everything else. Discard warmup explicitly rather than trusting it to wash out — with percentiles it does not wash out, it becomes your tail.

Open loop, or you’ll measure a lie

This is the one that quietly ruins load tests. A closed-loop generator — N workers that each send a request, wait for the response, then send the next — cannot generate load faster than the server responds. When the server slows down, the generator slows down with it, so the requests that would have arrived during the stall never arrive and never get counted. The result is a latency distribution that looks fine precisely when the system is failing.

This is coordinated omission, and it makes an overloaded server look healthy. Generate load open-loop instead: requests are issued on a schedule that does not care whether previous ones finished, and latency is measured from intended send time. On a box run near saturation — which is every configuration in this series — the difference between these two methodologies is larger than most of the optimizations the series will test.

Measure at the client

Server-side timers exclude queueing, transport, and detokenisation. The tenant experiences all three.

Reach steady state, and hold it

A batching server’s behavior at 30 seconds and at 10 minutes are different systems — the KV pool fills, prefix caches populate, the batch composition settles. Part 0 committed to sub-10-minute runs so results survive a --max-run-duration cutoff; those runs still need to reach steady state before the measurement window opens.

Reading a trace

The TPU profiler is how you tell why a number moved rather than just that it did — where the gaps in a step are, whether the host starved the device, whether a recompile crept in.

Written from a real profile, not yet

This section is deliberately empty. Everything above can be specified from first principles and holds regardless of what the hardware does; a walkthrough of a real trace cannot be, and writing one from memory would be exactly the kind of confident assertion this post argues against.

It gets filled in from the first captured profile, alongside Part 3’s material on where idle time actually comes from. Flagging the gap is more useful than papering over it.

The scoreboard

Metric	Answers	Reported as	Watch for
TTFT	How long until the tenant sees anything?	per-tenant p50 / p95 / p99	measured from prefill instead of arrival, hiding the queue
Inter-token latency	How fast does the text appear?	per-tenant p50 / p95 / p99	widening as batch size grows, even as throughput improves
Throughput	What is the box actually producing?	tokens/sec at concurrency C	quoted without a concurrency or a latency bound
HBM headroom	How close is the pool to binding?	free KV blocks, evictions/min	nothing — this one moves first, which is the point
MFU	Did this change move us toward compute-bound?	percent, compared against itself	being read as an absolute grade rather than a delta
Admission rejections	Who never got served at all?	count, per tenant	absent from the dashboard entirely, which is common

The last row is the one most easily forgotten. A rejected request has no latency, so it appears in no latency metric — and a system that stays fast by refusing half its traffic will look excellent on every other line of this table.

Part 2 is next: the architecture these numbers describe. The full request lifecycle on one machine — ingress, admission, engine, chip, and back out — and why the host CPU is a more common bottleneck than the accelerator everyone is watching.