The Machine in Motion

Part 2 drew the static map and said explicitly that it was saying nothing about what happens when the engine is cold, when a tenant bursts, or when a decode step blows past its deadline. This is that post.

The distinction matters more here than it would on a cluster, because most operational advice for serving systems is secretly advice about having more than one machine. Rolling restarts, canaries, draining a bad node, failing over — none of those exist here. Whatever goes wrong has to be absorbed by the box it happened on.

A cold engine is not a serving machine

Starting the process is not the same as being ready, and the gap is minutes, not seconds:

Loading ~26.7 GB of weights into HBM, from wherever they live
Compiling — XLA emits code per shape (Part 1), so every shape bucket you intend to serve has to be compiled before it’s served, or the first request of each bucket pays for it
Allocating and warming the KV pool

This is a recurring cost rather than a one-off. Part 0 put the series on flex-start, where a block ends at a --max-run-duration you set and the next one starts on a freshly created VM. Every session pays warmup again, which is the real argument for scripted provisioning — not the convenience, the fact that you are buying warmup repeatedly out of a 48-hour budget.

Health checks that lie

A health check that returns OK when the process is up but the engine is cold will route real traffic into a machine that cannot serve it. The first tenants get compilation latency attributed to their requests, and your p99 records something that has nothing to do with steady-state behavior.

Readiness should mean warm: weights resident, shape buckets compiled, a synthetic request completed. On a single machine there is no second replica to absorb the traffic while this one lies, so the lie is expensive.

The knobs that actually exist

“Autoscaling” here does not mean replicas. The control surface is smaller, and it splits along a line that matters operationally:

Knob	What it trades	Adjustable live?
Max concurrent sequences	throughput against inter-token latency	yes
Admission rate limits	acceptance against protection of in-flight	yes
Per-tenant KV budget	isolation against utilization	yes
Max sequence length	long-context support against pool pressure	yes
Quantization	quality against capacity	no — restart
Model or shard layout	everything	no — restart

The last column is the one to read carefully. A knob that requires a restart is not an operational control on this machine — it is a redeployment, and given the warmup above, it’s minutes of total outage with no replica to serve meanwhile.

So Part 4’s quantization decision, which reads like a tuning parameter, is really a deployment-time commitment. You pick it, you live with it until the next planned restart, and “we’ll adjust it if we need more pool” is not an available move at 3am.

Choose the degradation ladder, or the machine will

When load exceeds capacity, something gives. The only question is whether you decided what.

The undecided version is the worst one: queues grow, every tenant’s latency rises together, everyone breaches SLA simultaneously, and clients start timing out and retrying — which adds load, which is precisely the wrong response. Uniform degradation sounds fair and is the outcome nobody would choose.

A ladder is a set of steps taken in a stated order:

Stop admitting the lowest tier. Tiered tenants exist so that there’s something to shed first.
Cap max_tokens on new requests. Shorter responses free blocks sooner and lift pool pressure without refusing anyone.
Reduce max concurrency. Smaller batches, better latency for those admitted, less accepted work.
Reject with a clear, fast error. Part 1 counted admission rejections as a first-class metric precisely so this rung is visible rather than hidden.

The ordering encodes what you value, which is why it deserves to be written down somewhere other than in the emergent behavior of a queue.

Heterogeneous tenants are an asset

Here is the one genuinely cheerful thing in this post.

Tenants do different things. One sends 30k-token prompts and wants a short summary; another sends two sentences and wants a thousand tokens back. The first is prefill-heavy — compute-bound, matrix-shaped. The second is decode-heavy — memory-bound. From Part 3, those consume different resources.

Which means mixing them fills the machine better than serving either alone. A workload of purely decode-heavy tenants leaves the array idle; purely prefill-heavy tenants leave bandwidth unused. Heterogeneity, normally a scheduling nuisance, is here a bin-packing opportunity.

The mechanism that makes it work is chunked prefill.

What a long prefill does to everyone else

One tenant's large prompt occupies the engine. Unchunked it stalls every other tenant's token stream; chunked it interleaves and nobody notices.

An unchunked prefill takes the engine for as long as it needs, and every other tenant’s token stream stops dead for the duration. One tenant’s large prompt becomes every tenant’s latency spike — which is exactly the cross-tenant interference this whole series is trying to prevent, arriving through a door that has nothing to do with the KV pool.

Chunked prefill splits the prompt into slices sized to run alongside decode work. The big prompt takes marginally longer; nobody else notices it happened. It is the clearest example in the series of a scheduling decision converting a shared-fate problem into an isolated one.

Failure modes, and the asymmetry

The thing that makes single-machine operations different

On a cluster, most failures are partial. Here, the engine process holds every tenant’s state. An OOM does not degrade one tenant — it takes the process down and every tenant with it.

That asymmetry is the strongest argument for conservative admission control. The expected cost of refusing a request you could have served is a rejected request. The expected cost of accepting one you couldn’t is total outage. Those are not comparable, and admission thresholds should be set as though they aren’t.

The others are more ordinary, and the series has already named their signatures:

Host starvation — device gaps aligned with host activity (Part 2)
Eviction thrash — busy chip, no progress, healthy throughput, ruined percentiles (Part 6)
Recompilation — latency spikes at shape boundaries (Part 1)
Planned termination — flex-start’s --max-run-duration firing, which is not a failure but is an outage if you haven’t built draining

That last one is worth dwelling on because it’s scheduled. You know when the block ends. Draining — stop admitting, let in-flight sequences finish, checkpoint results off-box — is entirely achievable, and skipping it means losing whatever was in flight along with the benchmark run that was measuring it.

What the hardware has to settle

Actual warmup time end to end, which sets how much of each session is overhead
Whether chunked prefill’s chunk size can be tuned to hide prefill entirely, or only reduce it
What the degradation ladder does to worst-tenant p99 versus doing nothing
How close admission can safely run to the pool limit before OOM risk becomes real — the asymmetry above says stay well back, but “well back” is a number

Part 9 was meant to close the series with before-and-after benchmarks and a reference architecture. It needs a TPU, and the honest thing is to say so rather than produce numbers this series has spent nine posts refusing to invent. What exists instead is a stack of specific, falsifiable predictions — about tensor parallelism, about quantization, about where speculative decoding stops paying — waiting for a machine to be right or wrong about.