Toward Deploying a Multi-Tenant System on a Single TPU

The premise

Most writing about LLM serving quietly assumes an escape hatch. Latency climbing? Add replicas. Queue backing up? Scale the deployment. Out of memory? Bigger node. The advice is real, but it answers a question about procurement, not about engineering.

This series removes the escape hatch on purpose.

One machine. One HBM pool. One scheduler. Many tenants. No horizontal scaling.

Everything that would normally be solved by adding hardware has to be solved by policy instead — by how you budget memory, order requests, admit or reject load, and decide who gets evicted when the box is full. That constraint is not artificial. It is the situation for anyone on a fixed accelerator allocation, anyone on a quota-limited TPU reservation, anyone whose infrastructure budget is a number rather than a slider. It is also, not incidentally, the situation for anyone learning this on their own money.

The interesting claim of this series is that the single-machine constraint doesn’t make multi-tenancy a smaller problem. It makes it a purer one. When you can’t add nodes, every question becomes a question about the machine.

What “multi-tenant” means here

Multi-tenancy is an overloaded word. In most deployment writing it means isolation at the infrastructure layer: separate pods, separate namespaces, separate nodes, network policy between them. That is a solved problem and a boring one.

On a single TPU it means something much sharper. Every tenant’s request lands in the same scheduler queue, gets its KV cache allocated out of the same HBM pool, and competes for the same MXU cycles as every other tenant’s request. There is no hardware boundary between them. There is nothing between them at all except the policies you write.

So the questions get concrete fast:

If tenant A submits a 30k-token prompt, how much of the shared KV budget are they allowed to take, and what happens to tenant B’s in-flight decode when they take it?
If the KV pool is full and a high-priority request arrives, whose blocks get evicted?
If tenant C sends a burst, does tenant D’s p99 move? By how much? Is that acceptable, and who decided?
When you promise a tenant an SLA, what mechanism actually enforces it — or is it just a number in a contract that the scheduler has never heard of?

None of these have hardware answers on one box. All of them have software answers. That’s the series.

The machine, precisely

Vague hardware makes vague conclusions, so here is the exact target — and the exact reasoning, including the budget, because the budget shaped the design.

The chip: TPU v6e (Trillium)

Property	Value
Peak bf16 compute	~918 TFLOPs / chip
HBM capacity	32 GB / chip (~28.7 GiB usable)
HBM bandwidth	~1638 GB/s
On-demand price	~$2.70 / chip-hour (us-east1, us-east5)
Spot price	~$1.40 / chip-hour
Flex-start (DWS) price	~$1.35 / chip-hour

Two of those rows deserve a note now, because both cost me an assumption.

Usable HBM is not capacity HBM. The chip advertises 32 GB; roughly 28.7 GiB of it is actually available to your model and KV cache once the runtime has taken its share. On a machine where the KV pool is the binding constraint, a ~3 GB discrepancy is not a rounding error — it’s most of a tenant.

On v6e, spot is more expensive than flex-start. This reverses the pattern from v5e and from GPUs generally, where spot is the cheap tier. More on that below, because it changes how the series is actually run.

v6e is the newest TPU generation with a published price. Its successor, Ironwood (v7), had no public per-chip rate at the time of writing, which makes it impossible to plan a fixed-budget project around — a small but real illustration of the constraint this series is about.

The slice: `v6e-1` — one chip

This is the part that surprises people, so let me defend it directly.

The entire multi-tenant problem — shared KV pool, shared scheduler, contention, eviction, admission control, fairness, SLA enforcement — is fully present on a single chip. Every question in the list above is reproducible on 32 GB of HBM. Adding chips adds capacity; it does not add any new kind of problem. It mostly makes the existing problems less visible, which is the opposite of what you want while learning.

One chip also means contention shows up immediately instead of hiding behind headroom. That’s a feature.

It’s worth being straight about one thing: Google’s own documentation calls v6e-1 “primarily intended for testing,” and points at v6e-8 as the configuration “optimized for an inference use case.” That’s fair advice for someone shipping a product, and it is precisely why v6e-1 is the right choice here. The recommended configuration is recommended because it gives you enough headroom that these problems stay quiet. This series is about the problems.

What a single chip genuinely cannot demonstrate: tensor parallelism and ICI interconnect behavior. Those need real neighbors. So the series makes one deliberate exception — a few hours on a v6e-4 (4 chips, 128 GB, half-host VM) for the sharding and interconnect material, and for the final end-to-end numbers.

The budget

Line item	Config	Rate	Time	Cost
Main development & benchmarking	`v6e-1`, flex-start	~$1.35/hr	~48 hr	~$65
Tensor parallelism & ICI posts	`v6e-4`, flex-start	~$5.40/hr	~2.5 hr	~$14
Reserve — on-demand fallback, storage, overruns	—	—	—	~$21
Total				~$100

That is the whole hardware budget for the series. Not a rounding error on a research cluster — a real ceiling, and one of the parameters of the design.

Forty-eight hours is less than it sounds like and more than it sounds like. It is not enough to explore idly, which is why Parts 3 and 4 have to be fully drafted — down to which measurement answers which question — before the v6e-4 is booked at all. But benchmark runs are minutes, not hours, and most of the work in this series is reading traces and writing policy, both of which happen with the TPU switched off. The constraint is real and it is survivable, which is roughly the thesis of the whole series.

Everything runs on flex-start, and that sentence began life as “everything runs on Spot.” The correction is worth showing rather than hiding, because it is the kind of thing a fixed budget catches and a large one never would.

The reflex, carried over from GPUs, is that spot is the cheap tier: you accept preemption in exchange for a steep discount. On v6e it isn’t the cheap tier, and the structure of Google’s published pricing is the clearest way to see why.

DWS Flex-start is exactly half of on-demand, in every region and every TPU generation. Trillium $2.70 →$ 1.35, v5p $4.20 →$ 2.10, v5e $1.20 →$ 0.60, Ironwood $12.00 →$ 6.00. It is a flat 50% discount, and unlike on-demand — which varies from $2.70 in us-east1 and us-east5 to$ 2.97 in europe-west4 and $3.24 in Tokyo — the flex-start rate does not move between regions at all. A 50% preemption-free discount that is regionally uniform is a hard tier to beat.

Spot is the tier I can’t quote you. Google’s TPU pricing page lists Spot/Preemptible as a usage type and notes only that “Spot prices are dynamic and can change up to once every 30 days” — there is no published spot rate for TPUs in the regional table. Third-party readings of the Cloud Billing Catalog put v6e spot at roughly $1.40/chip-hour, which would make it more expensive than flex-start as well as reclaimable, but I could not confirm that against a Google source and am not going to assert it as though I had.

The recommendation doesn’t depend on that number. Flex-start is preemption-free and costs half of on-demand; spot would have to beat that and be worth its preemption risk. On a budget where a lost run costs measurement hours, it isn’t close.

What you give up with flex-start is not continuity but immediacy. You request a duration, the request queues until capacity exists, and you get an uninterrupted block that terminates on a --max-run-duration you set in advance. You wait at the front instead of getting evicted in the middle. Both are still the same silicon at the same clocks — there is no throttling and no performance penalty on either tier, so a measurement taken here is exactly as valid as one taken on-demand. You are buying a scheduling concession, not a performance discount.

Two consequences shape how this series is run — the same two as before, for a different reason:

Provisioning is a script, never a manual sequence. VM creation, vLLM install, model download, and benchmark harness setup are one command. The clock starts when the block is granted, not when you’re ready to use it, so any minute spent hand-installing vLLM is a minute of measurement you bought and didn’t take.
Runs are short and results are checkpointed off-box. Benchmarks are structured as many sub-10-minute runs writing results to Cloud Storage as they complete, not one long soak. --max-run-duration terminates the block whether or not you’re finished, and a soak that hasn’t written anything when the timer fires has produced nothing. This is better experimental hygiene anyway.

The reserve line exists for the case the queue doesn’t cooperate: capacity, not preemption, is the real risk. v6e can simply be unobtainable in a region for stretches, and a flex-start request that never gets scheduled is indistinguishable from not having a TPU. Holding ~ $21 back means the final Part 9 numbers can be bought on-demand at$ 2.70 rather than waiting on the queue — about eight hours of fallback, which is enough for one clean end-to-end run.

Two more practical notes. Quota, not price, is usually the blocker — request TPU quota in your target region before planning anything; the tiers are tracked separately. And check the current rates yourself. The on-demand and flex-start figures above were read off Google’s Cloud TPU pricing page in September 2026; regional variation applies to on-demand but not to flex-start, and the relationship between tiers is exactly the sort of thing that quietly changes.

The model: Gemma 4 12B

Property	Value
Parameters	11.95B — 48 layers, 262K vocab, 256K context
Usable HBM on `v6e-1`	~28.7 GiB
Weights + runtime (bf16)	~26.7 GB (~24.9 GiB)
Left for KV cache	~3.8 GiB (~4 GB)
Same model at FP8	~13.4 GB (~12.5 GiB) — leaving ~16 GiB of pool

(Vendor figures come in GB, HBM budgets get reasoned about in GiB, and the ~7% gap between them is large enough to matter when the whole pool is four gigabytes. Both units are given wherever the subtraction depends on it.)

Where the 32 GB actually goes

One v6e chip, one model, two precisions. The bar is the ~28.7 GiB the chip actually gives you — and at bf16 almost all of it is spoken for before the first request arrives.

Gemma 4 12B leaves roughly four gigabytes of KV pool on a single v6e chip. That is enough to hold a handful of concurrent tenants — and, crucially, few enough that you will run out. A model that never exhausts its KV budget teaches you nothing about multi-tenancy. Here the pool is the binding constraint from the first request onward, which is exactly what makes admission control, eviction policy, and per-tenant KV budgeting (Parts 5 and 6) load-bearing rather than theoretical.

It also sets up Part 4 with unusual clarity. Look at the last row of that table: the same model in FP8 occupies ~13.4 GB instead of ~26.7 GB, which takes the KV pool from ~3.8 GiB to ~16 GiB — a four-fold increase in the KV budget, bought with no additional hardware and paid for in accuracy. That is not an optimization footnote; on this machine it is the single largest lever available. Whether four times the pool also means four times the tenants is precisely the sort of claim this series should measure rather than assert — which is what Part 4 is for.

One caveat I’d rather flag now than discover in Part 5: this model has a 256K-token context window and about 4 GB of pool to serve it from. A single sufficiently long request can plausibly consume the entire budget on its own. Whether that turns out to be the series’ best illustration of tenant isolation or an admission-control problem that has to be solved before anything else works, is a question Part 1 will have to measure before Part 5 can lean on it.

Update, once the model’s configuration was actually checked: that caveat was correct, and more so than intended. Gemma 4 12B interleaves 40 sliding-window layers with 8 full-attention layers whose head dimension is double the others, so the uncapped part of the cache costs 128 KiB per token. A full 256K context needs roughly 32 GiB of KV — more than this entire chip has. The pool holds about 8,900 tokens in total across all tenants at bf16, and around 12 sequences at most. Part 5 works through the arithmetic, including retracting an intermediate guess of mine that had this backwards.

The stack

vLLM on TPU throughout. PagedAttention, continuous batching, and prefix caching are the mechanisms the whole series pulls on, and vLLM has first-class TPU support. Where vLLM’s default policy is insufficient for multi-tenancy — and it will be, repeatedly — that gap is the subject of the post.

The objective

A deployment can only be judged against a stated objective, so here is the one this series optimizes for.

Maximize the number of tenants served within their SLA, per dollar of fixed hardware.

Three parts, each load-bearing:

Within their SLA. Not average latency. A deployment where nine tenants are delighted and one is timing out has failed, and an average will hide that completely. Everything is measured at percentiles, per tenant.

Per dollar. The hardware cost is fixed and known ($100). Every optimization is judged by how many more tenants it buys on the same silicon — not by how much faster one request got.

Fixed hardware. No configuration in this series is allowed to conclude “and then add a node.”

The metrics that implement this objective get defined properly in Part 1, but the headline set is:

TTFT (time to first token) and inter-token latency — measured per tenant, at p50/p95/p99, never as an average
Throughput — tokens/sec and requests/sec at a given concurrency
MFU (model FLOPs utilization) — how much of the chip you’re actually using
HBM headroom — how much KV budget remains before eviction starts. On one machine this is the number. It is the single-machine metric that most writing on this topic doesn’t have, because on a multi-node cluster it never binds.

Roadmap

Part 1 — Metrics & how to measure. Front-loaded deliberately, so every later post can show a measured result rather than an assertion. Percentiles, load generation, warmup, reading a TPU profiler trace, and HBM headroom as a first-class metric.

Part 2 — System design & architecture. The anchor post. Full request lifecycle on one box: ingress → admission → vLLM engine → chip → detokenize → stream. What lives on the host CPU versus the TPU, and why the host is a more common bottleneck than people expect.

Part 3 — TPU foundations. MXU, HBM, ICI, and how they differ from a GPU. Pallas kernels. Where idle time comes from. (The v6e-4 burst.)

Part 4 — Precision & quantization. bf16, fp16, fp8, and the calibration tradeoffs — framed as the primary way to buy back HBM and fit more tenants. (The v6e-4 burst.)

Part 5 — Memory & the KV cache. PagedAttention, prefix caching, and the central single-box question: how do you budget a shared KV pool across tenants so that one can’t starve another?

Part 6 — Batching, scheduling & tenant isolation. Where multi-tenancy stops being theory. Continuous batching, priority tiers, per-tenant rate limits, SLA-aware ordering, and eviction policy when HBM is full — all software policy over one hardware queue.

Part 7 — Model-level speedups. Speculative/multi-token decoding, MoE serving, early exit — each evaluated against the constraint that expert weights compete with KV cache for the same ~28.7 GiB.

Part 8 — Runtime behavior on one node. Warmup, health checks, graceful degradation. The autoscaling knobs you actually have: batch size, max concurrency, KV budget. Failure modes — OOM, host starvation, tail-latency blowups — and how the architecture absorbs them.

Part 9 — Putting it together. Reference architecture, before/after benchmarks against Part 1’s metrics, and the single-machine deployment checklist.

Why this is worth doing

Two reasons.

The obvious one: fixed hardware budgets are the normal case, not the exception. Quota is finite, reservations are finite, and “add a node” is frequently not an available move.

The less obvious one, and the real motivation: a constraint you can’t spend your way out of forces you to actually understand the machine. When adding replicas is off the table, you have to know where the HBM went, why the chip is idle, what the scheduler chose and why, and which tenant paid for it. That understanding transfers upward — a cluster is many single machines, and the person who knows what one box is doing is the one who knows what the cluster is doing.

Part 1 is next: the metrics, and how to measure them without lying to yourself.