Predictions, Before the Hardware Can Argue Back

This was supposed to be the post with the numbers in it — before-and-after benchmarks against Part 1’s metrics, a reference architecture, a deployment checklist. That post needs a TPU, and there hasn’t been one.

Writing it anyway would mean inventing results, which is the single thing nine posts have been careful not to do. So this is the other post that can honestly exist at this point: everything the series has claimed, collected somewhere it can be scored.

The value isn’t in being right. It’s that a prediction written down before the measurement can be refuted by it, and a prediction written afterwards cannot. Every one of these was made in a post that had no data, and they are recorded here in the form they were made.

First, the things to check before spending anything

Neither costs money or TPU time, and both change what other numbers mean. One is now done.

Answered — and it went against me

This was the highest-value open item, and it has now been checked. Gemma 4 12B does use sliding-window attention: 40 of its 48 layers, window 1024. But the 8 full-attention layers carry head_dim 512, double the sliding layers, so the uncapped part of the cache costs 128 KiB per token and a 256K context needs about 32 GiB — more than the whole chip has.

I had speculated the windowing might make Part 0’s long-context worry “substantially wrong”. It doesn’t. Part 0 was right and my correction to it was the mistake, which is recorded in Part 5 rather than quietly removed.

The pool holds about 8,894 tokens at bf16 across all tenants, ~12 sequences at the sliding-layer floor. Part 4’s 8.5× multiplier survives untouched — it is a ratio, and the same correction applies to both sides.

Cost of finding this out: one config file.

The second was the pricing, and it has now been checked against Google’s own pricing page rather than a third-party reading of it.

The budget holds. Flex-start is $1.35/chip-hour and on-demand$ 2.70 in us-east1 and us-east5, exactly as Part 0 assumed — so the 48-hour cut and the $100 ceiling stand. Better, the structure turns out to be more robust than a quoted figure: flex-start is a flat 50% of on-demand across every region and every TPU generation, and unlike on-demand it doesn’t vary regionally at all.

One claim didn’t survive. Part 0 argued that spot costs more than flex-start on v6e, at ~$1.40. Google’s pricing page publishes no spot rate for TPUs — only a note that spot prices move every 30 days — so that comparison rests on a third-party reading I can’t confirm. Part 0 now says so instead of asserting it. The recommendation is unchanged, because a preemption-free 50% discount doesn’t need spot to be expensive in order to win.

Two free checks, two corrections. Both were worth doing before booking anything, which is the argument for the rest of this page.

The standing predictions

Each of these was stated in a post, with a direction. Confidence is my own, and the last column is what I would accept as a refutation.

#	Prediction	Confidence	Refuted by
1	MoE is disqualified on `v6e-1` by memory alone — the 26B variant doesn’t fit at bf16 and leaves no pool at FP8	High — it’s arithmetic	The reported parameter count being wrong
2	Host starvation is the default explanation for low MFU, not the model or the kernel	High	A profile showing device gaps that don’t align with host activity
3	The shippable quantization gain lands meaningfully below the ~8.5× arithmetic	High	A config passing a real eval at or near 8.5×
4	Quantized KV is riskier than quantized weights, because its error is written once and read on every later step	Medium	Long-context evals showing KV quantization is the more forgiving of the two
5	Recompute beats swap for eviction, since prefill suits the MXU and PCIe is what the machine is worst at	Medium	A measured crossover where swap wins at realistic sequence lengths
6	Tensor parallelism on `v6e-4` is a modest latency play — per-layer collectives are latency-bound at batch 1 and eat much of the gain	Medium	A clean, large latency win that scales with chip count
7	Speculative decoding, MoE and early exit all measure below their published numbers here, because batching takes the surplus first	Medium	Any of the three matching single-stream published gains at production batch sizes
8	Early exit is the least valuable of the three, largely defeated by lockstep batch progress	Medium	A compacting scheduler making it pay for itself

Prediction 6 is the one I’d most like to be wrong about, and Part 3 said so at the time: a clean tensor-parallelism win would be a better outcome than the muddy one I expect. Prediction 4 is the one I’d least like to be right about, since a KV quantization that degrades long context takes back half of Part 4’s headline.

Questions with no prediction attached

These are open in the honest sense — I don’t have a direction, and guessing one now would be inventing a claim to look decisive.

Measurement and mechanism

The split between memory bandwidth and pipeline occupancy in decode’s inefficiency
Whether a hand-written Pallas kernel beats what XLA emits, and by how much
Actual end-to-end warmup time, which sets how much of each 10-minute session is overhead
Real fragmentation under paging — whether one partly-filled block per sequence is the whole cost

Policy and tuning

Prefix cache hit rate on realistic multi-tenant traffic, which decides whether it’s a major win or a rounding error
Whether block-seconds accounting is cheap enough to run at step frequency, or whether the host cost eats the fairness it buys
Where the batch-size sweet spot sits for Part 0’s objective, rather than for throughput
Whether chunked prefill’s chunk size can hide prefill entirely or only reduce it
How close admission can run to the pool limit before OOM risk becomes real

Isolation, which only adversarial testing answers

What a deliberately abusive tenant does to everyone else’s p99 under each fairness policy
Whether reservation-plus-burst holds under that tenant
Whether eviction thrash appears under realistic load, or only under hostile load
What the degradation ladder does to worst-tenant p99 versus doing nothing

That last group is the one I’d protect if the budget gets tight. Throughput numbers are easy to gather and easy to find elsewhere; a measurement of what one bad tenant does to the others is the thing this series exists to produce, and nobody publishes it.

How to score this

The rules, set now rather than later

Negative results get published. A prediction that fails is the most informative thing on this page, and quietly dropping one would make the whole exercise decorative.

The wrong ones get quoted back. Where a measurement contradicts a post, the post gets a correction that names what it said, in the style of the Part 5 correction against Part 0 — visible rather than silently edited.

“Roughly right” is not a pass. Prediction 6 says modest. If tensor parallelism turns out to give a clean 3× at batch 1, that is a refutation, not a near-miss, and it should be recorded as one.

Unmeasured stays unmeasured. If the budget runs out before something is tested, it stays on this page as untested. The series has marked every unmeasured claim so far and running out of money is not a reason to stop.

What this page is actually for

There is a failure mode in writing about performance where the author measures first, finds something interesting, and constructs the reasoning that predicts it. The result reads like insight and contains none, because the hypothesis was fitted to the data after the fact.

The only defense is to write the predictions down when you still might be wrong about them. Nine posts of reasoning from first principles produced eight claims with directions and thirteen questions without. That’s the ledger. The hardware gets to argue back whenever it’s booked, and this page is what it will be arguing with.

The series stands complete as far as reasoning alone can take it. What remains is the v6e-1 session itself — and the real Part 9, with the benchmarks in it, once there is something to benchmark.