Quantization Is How You Buy Tenants

On a machine you can scale, quantization is an optimization: the same work, cheaper. On this machine it is something else. Part 0 put the KV pool at roughly 3.8 GiB after weights — a few tenants before the pool binds. Quantization is the only lever that changes that number without changing the hardware.

So this post isn’t about making the model faster. It’s about how many tenants fit, and what you pay in output quality to fit them.

Why bf16, and not fp16

Both are sixteen bits. They spend those bits differently, and the difference decides which one a TPU wants.

bf16 keeps fp32’s 8 exponent bits and cuts the mantissa to 7. Same dynamic range as fp32, less precision within it. fp16 uses 5 exponent bits and 10 mantissa: more precise, dramatically less range.

For neural networks that trade is lopsided, because the thing that breaks a network is a value falling outside representable range — an activation overflowing to infinity or underflowing to zero — not a value being slightly imprecise. fp16’s narrow range is why fp16 training needs loss scaling, an entire mechanism whose only job is keeping values inside the window. bf16 doesn’t need it, because its window is fp32’s.

The TPU makes the choice for you anyway: the MXU takes bf16 inputs and accumulates in fp32 natively (post 2, §3.2). fp16 is not the path the hardware is built around, and for inference there is no argument that recovers it. bf16 is the baseline here, and the interesting question is what’s below it.

Two levers, not one

Here is the thing most treatments of this conflate, and Part 0’s memory table quietly did too: quantizing weights and quantizing the KV cache are separate decisions.

Two levers, multiplied

Weight quantization grows the pool. KV quantization shrinks what each tenant puts in it. They are independent, and they compound.

They work on opposite sides of the same equation. Quantizing weights makes the pool bigger — every gigabyte the weights give up becomes pool. Quantizing the KV cache makes each tenant’s footprint smaller — the pool is unchanged, but more fits in it.

Because one grows the container and the other shrinks the contents, they multiply. Weights alone is about 4.3×. KV alone is about 2×. Together, roughly 8.5× the tokens the pool can hold, on identical silicon.

The baseline turned out tighter than assumed

Written before the model’s attention configuration was confirmed. It since has been, and the numbers are worse than this post assumed — the pool holds about 8,894 tokens at bf16, not the comfortable few thousand implied above. Part 5 has the arithmetic.

The multipliers survive unchanged, because they are ratios and the same correction applies to both sides: 8,894 tokens at bf16 becomes 17,788 with FP8 KV, 37,917 with FP8 weights, and 75,835 with both — still 8.5×. What changes is the stakes. At a baseline this tight, quantization stops reading like the largest available optimization and starts reading like a precondition for serving more than a dozen tenants at all.

That is arithmetic, not a result

Every number above follows from Part 0’s memory table by division. None of it has been run. The capacity side of quantization is the easy half precisely because it is arithmetic — you can predict it exactly. The half that decides whether any of it is usable is the accuracy cost, and that cannot be derived, only measured.

What FP8 actually involves

FP8 is two formats, not one, and the distinction matters.

E4M3 — 4 exponent bits, 3 mantissa — favors precision over range, and is the one used for weights and activations in inference. E5M2 trades mantissa for range and shows up mostly in training, where gradients span far more orders of magnitude. For this series, E4M3 is the relevant format.

The hard part isn’t the format, it’s the scale factors. Eight bits cannot span the range a tensor occupies, so each tensor gets a scaling factor mapping its actual range onto what FP8 can represent. Choosing those factors is calibration, and there are real decisions in it:

Granularity. One scale per tensor is cheapest and crudest. Per-channel or per-row costs more bookkeeping and handles uneven tensors far better.
Static or dynamic. Static scales are computed offline from a calibration pass over sample data. Dynamic scales are computed at runtime from the actual tensor — more accurate, and directly in the decode path this series is trying to keep short.
Outliers. LLM activations are known to have a few channels with values far outside the rest. One outlier drags a per-tensor scale wide enough that everything else loses resolution — which is the main argument for per-channel scaling, and for leaving particular layers unquantized.

That last point is why quantization rarely ends up uniform. The realistic outcome is a mixed configuration where most of the model is FP8 and a few sensitive tensors aren’t, and finding which ones is empirical work.

The cost, and how not to lie about it

Capacity is arithmetic. Quality is not, and this is where the discipline from Part 1 has to apply or the whole lever becomes a guess.

The failure mode to avoid

“I quantized it and the outputs looked fine” is not a measurement. It’s a handful of samples, chosen by someone who wants it to work, evaluated by the person who did it. It is exactly the kind of claim Part 1 exists to rule out — and it’s more tempting here than anywhere else in the series, because the capacity win is so large and so easy to compute.

Two things worth knowing about measuring it:

Perplexity is not enough. It’s cheap and it’s the default, and it’s insensitive to the failure modes that actually matter — a model can hold near-identical perplexity while getting noticeably worse at instruction-following, long-context recall, or structured output. Whatever the tenants actually do is what has to be evaluated.

KV quantization and weight quantization fail differently, and I’d expect KV to be the riskier of the two on this model. Quantized KV entries are written once and then read back on every subsequent decode step, so error introduced early persists for the entire sequence rather than being re-derived. On a model with a 256K context window that’s a long time for a small error to matter. I want to flag that as a suspicion rather than a finding — it’s the kind of thing that either shows up clearly in a long-context eval or turns out not to matter at all, and I don’t know which yet.

Where sharding comes in

Scale factors are per-tensor, and tensor parallelism splits tensors across chips. So a sharded quantized model has to either compute scales per shard — cheap, slightly less accurate, since no chip sees the full tensor’s range — or communicate them, which adds to the per-layer collective traffic Part 3 already predicted would be the expensive part.

That interaction is worth naming and not worth resolving here. Part 3 owns whether tensor parallelism earns its keep at all; if the answer there is no, this question is moot.

What the hardware has to settle

Part 0 required Parts 3 and 4 drafted before booking any v6e-4 time. This completes that. The questions this post hands to the first real session:

Whether FP8 weights hold quality on the tasks tenants actually run — measured on a real eval, not perplexity and not vibes
Whether quantized KV degrades long-context behavior, which is the risk I’d bet on and would like to be wrong about
Which tensors, if any, have to stay at higher precision, and what that costs back in HBM
Whether the ~8.5× arithmetic survives contact with a configuration that’s actually acceptable to ship

The honest expectation is that the shippable number is meaningfully below 8.5×. Finding out where it lands is the point.

Part 5 is next, and it’s the one this series has been building toward: the KV cache itself. PagedAttention, prefix caching, and the question that only exists on one machine — how do you budget a shared pool across tenants so that one of them cannot starve another?