The anchor post of the series — the full request lifecycle from ingress to streamed token, which stages run on the host CPU versus the TPU, why the host starves the accelerator more often than anyone expects, and where admission control has to sit.
Opening a series on serving an LLM to many tenants from one TPU chip — one HBM pool, one scheduler, a fixed $100 budget, and no horizontal escape hatch. The machine, the model, the objective, and what the constraint is for.
Why the KV cache dominates memory on one chip, how PagedAttention makes the pool accountable, why cross-tenant prefix caching is a security question and not just a performance one, and the design space for budgeting a shared pool.
The scoreboard for the rest of the series — the two latencies, throughput and what it hides, why every number is a per-tenant percentile, and HBM headroom as the one metric that only binds on a single machine.