Three Speedups That All Want the Same Thing
On this page5
Every technique in this post is presented elsewhere as a straightforward win. Each one asks for something, and on this machine there is exactly one place it can come from: the pool that Part 5 established is already the binding constraint. A speedup that costs a gigabyte costs tenants.
There’s also a subtler shared cost, which won’t be obvious until the end, so it’s worth stating up front: all three of these exploit surplus that batching also wants.
Speculative decoding, and why it suits this hardware
Part 3 established that decode leaves the chip’s arithmetic capacity mostly idle — every weight read from HBM, used once, the array mostly draining. There is enormous surplus compute and no memory bandwidth to spare.
Speculative decoding is the technique shaped exactly like that imbalance.
Speculative decoding, one round
A cheap draft proposes several tokens ahead. The target model then verifies the whole proposal in one forward pass — and because decode is memory-bound, checking four candidates costs barely more than decoding one, since the weights get read either way. Accepted tokens are kept, the first rejection discards the rest, and the round advances by however many survived.
What it actually trades
Speculative decoding spends idle FLOPs to buy fewer HBM reads. That is precisely the exchange rate decode makes favorable — you are paying in the resource decode leaves lying around and saving the one it is starved of.
Of the three techniques here, it is the only one whose mechanism is aimed at decode’s specific imbalance rather than at general inefficiency.
The costs are real though, and two of them bite harder here than in most write-ups.
A draft model needs HBM. A separate small model has to be resident, and on a ~3.8 GiB pool it comes directly out of tenants’ KV budget. This is the thing that makes MTP — multi-token prediction heads on the target model itself — the better shape for this machine. Extra heads are far smaller than a second model, and there’s no separate set of weights to keep resident.
Acceptance rate decides everything. A proposal that’s usually rejected has spent draft compute and verification compute to advance one token, which is worse than not speculating. Acceptance depends on how well the draft tracks the target, and it is workload-dependent, so it is measured rather than assumed.
MoE: the wrong trade on this machine
A mixture-of-experts model routes each token to a few of many experts. Compute per token stays low — only the selected experts run — while all experts must be resident in memory, because any token might route anywhere.
That is a trade of memory for compute. On this machine it is exactly backwards: memory is the binding constraint and compute is the thing sitting idle.
The arithmetic is unkind. Gemma 4’s MoE variant is reported as 26B total with about 4B active. At bf16 that’s roughly 52 GB of weights — against ~28.7 GiB of usable HBM, it does not fit at all. At FP8 it’s ~26 GB, which fits with almost nothing left for a KV pool, which is to say it doesn’t fit either, for any purpose this series cares about.
A multi-tenant effect specific to MoE
There’s a second problem that only appears once tenants are mixed. Expert routing has locality: a batch of similar requests touches a narrow set of experts. But multi-tenant batching deliberately mixes tenants doing different things, so a batched step touches a much wider spread of experts than any single tenant would alone.
On a machine where experts must all be resident, that costs less than it would elsewhere — but it undermines every optimization that depends on expert locality, and it gets worse precisely as you batch more tenants together for throughput. I have not seen this tension discussed, and I would want to measure it before trusting any MoE serving result gathered on single-tenant traffic.
For a single v6e-1, MoE is effectively disqualified by memory before any of this matters. It’s worth understanding anyway, because it’s the clearest case of a technique whose entire value proposition inverts under this constraint.
Early exit, and what batching does to it
Early exit stops computing at layer k when the model is already confident, skipping the rest of the network.
The mechanism is sound and the interaction with batching is brutal. A batch runs its layers in lockstep — every sequence in the step advances through layer 20 together. A sequence that could have exited at layer 20 in a batch where others need all 48 saves nothing at all, because the step runs to 48 regardless. The saving only materializes if the scheduler can compact the batch mid-network, which is significantly more machinery than early exit is usually presented with.
So early exit’s benefit is roughly inversely proportional to batch size — and this series batches as hard as it can, for all the reasons in Part 6. Of the three, it’s the one I’d expect least from here.
The thing all three have in common
| Technique | Spends | Costs from the pool | Competes with batching for |
|---|---|---|---|
| Speculative decoding | idle compute | draft weights (MTP: little) | idle compute |
| MoE | memory | all experts, resident | memory |
| Early exit | nothing directly | nothing | lockstep batch progress |
Read the last column. Each of these techniques harvests something that batching also harvests, and batching gets there first.
Speculative decoding exploits idle compute — but a large batch has already filled the array, so there is less idle compute to exploit. Early exit needs sequences to progress independently — but batching makes them progress in lockstep. MoE wants memory this machine does not have and batching wants to fill.
The expectation this leaves me with
Single-stream benchmarks are where these techniques look best, and a multi-tenant box run near capacity is the opposite of a single stream. My expectation is that the measured value of all three here is materially below what their published numbers suggest — not because the numbers are wrong, but because they were gathered in the regime where the surplus exists.
If that’s right, the useful finding is the crossover: the batch size above which speculative decoding stops paying. That number is far more actionable than any headline speedup, and it is specific to this machine and this model.
What the hardware has to settle
- MTP acceptance rate on realistic multi-tenant traffic, and whether it holds up when tenants are doing different things
- The batch size at which speculative decoding stops paying — the crossover above
- Whether MTP heads cost little enough HBM to be worth it against the KV they displace
- Whether early exit can be made to work with a compacting scheduler, or whether the machinery costs more than it saves
Part 8 turns to runtime behavior — the material Part 2 deliberately held back. Warmup, graceful degradation, the autoscaling knobs that exist when adding a node isn’t one of them, and the failure modes this architecture has to absorb rather than escape.