The Anatomy of a Request on One TPU Machine
On this page7
Part 1 built the scoreboard. This post builds the map — the thing every later part points at when it says “the scheduler” or “the queue” or “the host.”
It is deliberately static: what the components are, where they run, and how a request moves between them. How the system behaves under load — warmup, degradation, failure modes — is Part 8. Keeping those separate matters, because the most common mistake in writing about serving architecture is describing a runtime symptom as though it were a structural fact.
One request, end to end
One request, both planes
Count the stages. Eight, and two of them run on the TPU.
That ratio is the single most useful thing on this diagram, and it is the opposite of most people’s mental model. The accelerator is the expensive part, the interesting part, and the part everyone watches — and the request spends the majority of its wall-clock life being handled by a CPU.
Note also that the boundary gets crossed twice, and that the decode loop runs per token. A 500-token response is 500 trips around that loop, each one handing a sampled token back to the host to be turned into text and pushed down a socket.
What runs where
| Stage | Runs on | Scales with | Notes |
|---|---|---|---|
| HTTP/gRPC | host CPU | concurrent connections | one live connection per streaming tenant |
| Admission | host CPU | arrival rate | pure policy — the subject of Part 6 |
| Tokenize | host CPU | prompt length | single-threaded per request |
| Schedule | host CPU | batch size, KV pool state | one queue, all tenants |
| Prefill | TPU | prompt length | compute-bound |
| Decode | TPU | output length | memory-bound, one step per token |
| Detokenize | host CPU | total tokens/sec | runs at the machine’s whole aggregate rate |
| Stream out | host CPU | concurrent connections | slow clients cost host memory, not HBM |
The KV cache is the exception that doesn’t fit a row: the blocks live in HBM, but the bookkeeping — which block belongs to which sequence, what’s free, what to evict — is a host-side data structure. Part 5 is about the blocks. Part 6 is about the bookkeeping.
Where the API server lives
There are two arrangements, and the choice is more consequential on one machine than on a cluster.
In-process puts the HTTP server and the engine in the same process. No serialization, no IPC hop, and the lowest possible overhead per token — which is why it’s the default. The cost is that the server and the engine now contend for the same CPU, the same event loop, and in Python the same interpreter. A burst of connection handling can delay the scheduler step that feeds the TPU.
Sidecar splits them: an API process talks to an engine process over a local socket or shared memory. You pay a hop per request and some serialization, and you buy isolation — a slow client or a flood of connections degrades the API process without stalling the engine loop.
Why this is a single-machine question
On a cluster the answer is usually “in-process, and run more replicas” — the isolation you want comes from having separate machines. Here there is one host, one set of cores, and every process on it is competing with the thing that feeds the accelerator. The arrangement stops being a deployment detail and becomes a resource-allocation decision.
The host is the bottleneck more often than the chip
This is the section that surprises people, and it follows directly from the eight-stage count above.
Every stage on the host is CPU work happening while the TPU is either idle or waiting. Four of them scale in ways that get worse exactly when you’re succeeding:
- Tokenization scales with prompt length and is single-threaded per request. Long prompts — the multi-tenant norm, with system prompts prepended — spend real milliseconds on CPU before the accelerator sees anything at all.
- Detokenization runs once per token per request, which means it runs at the aggregate token rate of the whole machine. Batch more tenants to raise throughput and you have proportionally raised the detokenization load on a CPU that was not the thing you were optimizing.
- Sampling — turning logits into a chosen token — is small, frequent, and sits directly in the decode loop. Anything slow here delays every subsequent step.
- Host-to-device transfer over PCIe is orders of magnitude slower than HBM. Batched token ids are small, so this is rarely the wall, but it is on the critical path of every step.
The failure mode is specific and worth naming: the TPU finishes a decode step and waits for the host to prepare the next one. The chip is idle, MFU is low, and nothing about the model or the kernel is wrong. You are CPU-bound on an accelerator machine.
This is why Part 1 insisted MFU is a delta, not a grade
A change that should have raised MFU — bigger batches, better kernels — leaving it flat is the signature of exactly this. The accelerator can’t use the headroom you just gave it because it is waiting on a core. Reading MFU as an absolute score tells you the number is bad; reading it as a delta tells you the host is starving the device, which is a fixable, findable problem.
Backpressure and admission control
When the box is full, there are exactly three things you can do with an arriving request: run it, queue it, or refuse it. On a single machine there is no fourth option, and pretending otherwise is how systems fall over.
Queueing is not a solution, it’s a deferral. If arrivals outpace what the machine can serve, a queue converts rejections into timeouts — the same requests fail, but later, after consuming memory and a connection while they waited, and after degrading everyone ahead of them. An unbounded queue is a promise the machine cannot keep.
So the queue must be bounded, and the bound is a real decision: it should be roughly the number of requests that can complete within SLA at the current service rate. Beyond that, refusing quickly is kinder than accepting slowly. A tenant that gets a fast, explicit rejection can retry or shed load; a tenant that waits sixty seconds for a timeout has had sixty seconds of their capacity wasted too.
Where admission has to sit
There is a genuine tension in placing it, and it’s worth stating plainly because it has no clean answer.
Admission wants to happen early — before tokenization — so you don’t spend CPU on requests you’re going to reject. But the information admission needs most is the prompt’s token count, which determines how much KV pool the request will consume, and you don’t have that until after tokenization.
The practical resolutions are all compromises: estimate token count from byte length and accept the error, admit in two phases with a cheap gate before tokenizing and a real one after, or eat the tokenization cost and accept that rejected requests still cost CPU. Which one is right depends on how adversarial your tenants are, and there isn’t a universally correct answer.
Part 6 takes this up properly — priorities, per-tenant limits, and what happens to in-flight work when the pool is exhausted. For now the structural point is enough: admission control is the only mechanism that protects the tenants already running, and it lives on the host, ahead of everything expensive.
What this post deliberately isn’t
Everything above is the static map. It says nothing about what happens when the engine is cold, when a tenant bursts, when the KV pool fragments, or when a decode step blows past its deadline. Those are runtime behaviors, they have their own post, and describing them here would blur the one distinction that keeps both posts useful.
Part 3 is next, and it goes down a level: what the TPU is actually doing during those two device-plane stages, why decode is such an awkward fit for a systolic array specifically, and where the idle time in a profiler trace comes from. It assumes the TPU Foundations posts rather than repeating them.