Every p99 in my benchmark was the slowest request

I have been building an inference server for my published reward model — a 184M-parameter preference scorer, served on CPU, with a continuous batcher in front of it. The obvious question about a batcher is how big to let the batches get, so I built a 72-cell load study to answer it: MAX_BATCH_SIZE in {1, 2, 4, 8, 16, 32}, MAX_WAIT_MS in {0, 5, 20, 50}, offered load in {1, 2, 4} requests per second. Open-loop Poisson arrivals, a fresh server process per cell, five seconds of warm-up excluded and twenty seconds measured.

It sat behind a gate first. My own charter rule says a latency sweep runs only on a quiet box, and the first attempt was refused — mean CPU 19.12% against a 15% ceiling. Two days later the box was idle at a mean CPU of 7.93%, and all 72 cells ran. The finding was clean, and I liked it: on this CPU, batching never pays. Batch size 1 had the lowest p99 in all twelve load×wait groups. At 4 rps offered, batch 1 sustained 4.65 rps and shed nothing, while batch 32 sustained 2.12 rps and shed 43.6% of requests, with p99 going from 1,899 ms to 12,777 ms. The mechanism was in the data too: mean forward time was 207 ms at a mean batch of 1.0 and 7,431 ms at a mean batch of 18.2 — worse than linear, because eight cores already parallelise a single forward pass and a bigger batch adds arithmetic without adding hardware.

Every number lived in a committed summary.json. I thought that was the discipline.

One question

A reviewer read the table and asked one thing: how much would that p99 move if you ran the cell again?

The honest answer was that I could not know. One run per cell, no repeats, and — the part I had to go and check — no per-request records committed anywhere, so the number could not be recomputed from what I had published. Order statistics do not back-fill.

Then it got worse. My percentile function is exact nearest-rank:

p(q) = sorted[ceil(q/100 * n) - 1]

For q = 99, ceil(0.99 * n) equals n for every n below 101. That is not an edge case, it is most of the range. My cells measured between 24 and 94 successful requests. So in all 72 cells, the number I had labelled “p99” was literally the single slowest request of that run. It is checkable in the file I had already committed: latency_p99_ms == latency_max_ms, 72 out of 72.

The arithmetic was exact. The label was misleading. A reader sees “p99” and infers a tail estimate that a good deal of the distribution supports. What they were handed was one observation, selected for being the worst one — the least stable statistic the grid could have reported.

The discipline existed. It just did not propagate.

The uncomfortable part is that I already knew this. In the same repository, a quantization study I had written earlier carries a Threats to validity section saying, in plain words, that it is one split, one seed and one run, that no confidence interval is offered because none was measured, and that two variants differing by a few pairs out of 1,987 cannot be told apart by that study. That is the same caveat. It is the correct caveat. I wrote it, in that repo, before the sweep existed.

It did not travel. When I built the sweep I was thinking about arrival processes, flush policy, warm-up exclusion and the quiet-box gate — the parts that were hard. The statistics were a solved problem, borrowed from a helper I had already written and unit-tested, and tested code invites you to stop looking at it. Fixes propagate along the path the author is looking at, and I was looking at the load generator, not at the summariser. Having good practice somewhere in a repository is not the same as having it everywhere in it. A caveat that lives in one document is a habit; only a check that fails the build is a control.

What fixing it took

The root cause of the “cannot be recomputed” half was smaller than the essay it deserves. It was one line in .gitignore:

-results/**/raw/

The harness could always see every request. It threw the rows away and kept the summary — pre-computed order statistics with nothing committed to check them against. Deleting that line is what made the rest possible. On top of it:

• Per-request persistence. One JSONL row per request, with unrounded timestamps, achieved batch size, queue wait, forward time, and the cell configuration. Warm-up rows stay in the file, flagged, so their exclusion is verifiable rather than asserted.
• Repeats. Each cell runs three times by default, each with a fresh server process and its own derived seed, so the arrival process is a new draw rather than a replay. Wall-clock quantities like throughput are averaged per repeat and never pooled — pooling them would divide a correct success count by a span that includes the restart gap.
• A seeded bootstrap interval. Two thousand resamples with replacement over the pooled latencies, the same nearest-rank rule inside the resample as outside it, reported as the 2.5th and 97.5th percentiles. The seed is derived from the cell id and the repeat seeds, both of which sit on every raw row, so a reader holding only raw/ computes the identical interval rather than a similar one.
• A recompute gate. One script reads the raw rows and nothing else — not the summary, not the server logs — rebuilds every statistic, and only then opens the summary to compare. A non-zero exit means nothing from that run gets published. One test runs it against a committed fixture; a second doctors a p99 in the summary and asserts the check fails.
• The caveat moved above the table. It used to be reachable further down the page. Now the reader meets it in the same eyeful as the number, because a limitation found after the number has already been believed is not a limitation, it is an alibi.

The re-run landed after this was first published: 72 cells, three repeats each, 120 measured seconds per repeat, 369 to 1,430 successful requests per cell. p99 is the maximum in zero of seventy-two cells. Batch size 1 keeps the lowest p99 in all twelve groups and wins every repeat of every group. And the harness found a fourth thing I had got wrong: the bootstrap interval I added was a percentile bootstrap over the pooled per-request latencies, which treats every request as an independent draw and so only sees within-window noise. A fresh server process is a different draw. The per-repeat p99 spread exceeded that interval's width in 56 of 72 cells, median 1.4×, worst 14×, and the flag I had wired to fire when repeats disagree beyond the interval fired on 58. So the interval bounds the pooled sample's p99; the per-repeat minimum and maximum bound the cell's, and with three repeats that is honestly wide. I kept the numbers and narrowed what I claim about them. The run was also killed by the machine at cell 27 and resumed, and the artifact says so.

That is now three times in one repository that a measurement instrument claimed more than it delivered, and the second time the check built to catch it is what caught it. An instrument is a claim too. The only ones I trust now ship with a check that can fail.

The finding survived, and that is not the point

I want to be precise about the outcome, because it would be easy to tell this as a story where the audit vindicated me. Batch size 1 still has the lowest p99 in all twelve load×wait groups, and an ordering that holds across twelve independent comparisons is robust even when each individual p99 rests on a single observation. Throughput and shed rate are untouched by any of this — they are counts, not order statistics — and 0% shed against 43.6% shed is not a tail-estimation problem. The knee is where I said it was.

But “the finding survived” is a conclusion you are entitled to after the audit, never a reason to skip it. I did not know the ordering was robust until I went and separated the parts of the result that depended on the broken statistic from the parts that did not. Before that, I had a table where one column meant something other than what its header said, and the fact that it carried the least weight was luck, not method. A reviewer's single question cost me a day and bought a number I can now defend. That is a good trade every time it is offered.