TPU › Multi-Tenant LLM Serving
Deploying an LLM for many tenants on a single TPU machine — one HBM pool, one scheduler, and no horizontal escape hatch.
Opening a series on serving an LLM to many tenants from one TPU chip — one HBM pool, one scheduler, a fixed $100 budget, and no horizontal escape hatch. The machine, the model, the objective, and what the constraint is for.
The scoreboard for the rest of the series — the two latencies, throughput and what it hides, why every number is a per-tenant percentile, and HBM headroom as the one metric that only binds on a single machine.
The anchor post of the series — the full request lifecycle from ingress to streamed token, which stages run on the host CPU versus the TPU, why the host starves the accelerator more often than anyone expects, and where admission control has to sit.
Why decode is an awkward fit for a systolic array specifically, what Pallas is for, the three places idle time comes from on one chip, and an honest prediction about tensor parallelism that the hardware may well refute.
Precision as a capacity lever on a single chip — why bf16 and not fp16 on TPU, what FP8 calibration actually involves, and why weight quantization and KV cache quantization are two independent levers that multiply.
Why the KV cache dominates memory on one chip, how PagedAttention makes the pool accountable, why cross-tenant prefix caching is a security question and not just a performance one, and the design space for budgeting a shared pool.
Where the series stops describing hardware and starts writing policy — continuous batching, what fairness even means on one queue, priority tiers that do not starve, and how to pick an eviction victim without punishing your best tenant.
Speculative decoding, MoE serving and early exit, each evaluated against a fixed pool — why speculative decoding fits this hardware unusually well, why MoE is close to disqualified on one chip, and why all three compete with batching for the same surplus.
Runtime behavior on one node — why a cold engine takes minutes to become a serving machine, which knobs are operational and which are redeployments, choosing a degradation ladder before the machine chooses one for you, and why heterogeneous tenants are an asset.
Every falsifiable claim the series has made, collected into one scoreable page — what I expect, how confident I am, and exactly what would prove each one wrong.