What your benchmark cannot tell you

Most reported degradations are smaller than the measurement can resolve. Including one of ours.

State the noise floor before the delta

A 200-question benchmark carries roughly ±8 questions of run-to-run variation. That comes straight from the harness’s own reported standard error, doubled because you are comparing two independent runs. On HumanEval’s 164 problems it is about ±6.

Which means a three-question difference is not a small effect. It is not an effect. And a great many published “small but real” degradations sit comfortably inside their own error bars, unstated.

We got this wrong in public

We claimed refusal removal cost 1.5–4 points of instruction-following, on the strength of ten measurements that all pointed the same way. More data killed it: later pairs went the other direction, and one modified model scored +2.0 points on maths — which the process cannot cause. Ten same-signed results inside the noise band is a pattern you can talk yourself into. We did.

Report counts, not percentages

“78.0% versus 78.5%” reads like a gap. “156 versus 157 out of 200” obviously does not. Those are the same two results. The percentage hides how many questions were asked, and that is the one thing a reader needs in order to judge whether the difference means anything.

Two habits follow. Publish the denominator. And when a modified model beats its original, say so rather than quietly rounding — it is the clearest available evidence that you are reading noise.

Never compare across harnesses

The same weights, measured two ways:

Qwen3.5-0.8B, GSM8K strict
transformers + bitsandbytes 4-bit .... 17.0%
llama-server + GGUF Q4_K_M ........... 37.5%

Twenty points from the harness alone — far larger than the quantization effect we were trying to detect. Had we compared one against the other we would have concluded that four-bit quantization doubles mathematical ability.

The tell was in the strict-versus-flexible spread: one harness produced answers the strict parser could read, the other did not. Same model, same questions, different extraction.

Four habits that catch the rest

Validate a task on 20 problems before running 200. We lost two days to a code benchmark returning zeros.
Treat an exact 0.0 as a broken extractor until proven otherwise. It happened three times, and it was never the model.
Force greedy decoding explicitly. Some servers sample by default while your reference path does not. That produced a phantom seven-point drop.
Cap generation budgets. A checkpoint shipping max_new_tokens: 32768 will exhaust VRAM mid-run, hours in.

The payoff

With the floor stated, our headline becomes defensible rather than impressive: across 51 comparisons, refusal removal costs no capability this instrument can detect. That is a weaker claim than “it costs nothing” and a much stronger piece of evidence. See all 51.