Our SWE-bench harness scored the gold patch 0/25. None of our zeros measured the model.

Correction (2026-09-25, rev 3). The original headline overstated the evidence. The five 0/25 readings came from four model arms and the benchmark’s gold patch, all scored by one broken judge; earlier harness generations failed in different ways. In the gold run, 10 of the 25 failures were test IDs the runner could not collect (8 Django, 2 SymPy); 13 never ran a test in the host’s Python 3.12 environment; the remaining 2 IDs in our list are not SWE-bench instances at all. The 15.5× gain is TP2 + DFlash-64 (1,746.5 tok/s) over TP2 (112.7 tok/s) measured in a later session on a shared node, not from the 1,795.1 window; cross-session speedup spans 9.5–9.8×. The composite line now uses English thousands separators. The explanation of how the 10.1× came about was also corrected. See LN-002 for our second audit note.

Original headline: "We scored 0/25 five times. None of them was the model." (overstated; corrected in rev 3)retracted

On the fifth attempt we ran the benchmark’s own gold patch — the diff it declares correct — through the exact same generate/apply/test path as every model arm. All 23 gold patches applied; the other 2 IDs in our list are not SWE-bench instances and never materialized (apply rate 92%). Resolved: 0/25. File: swe_gold_sanity.json (n=25). After the judge and classifier fix, the same n=25 is in swe_gold2.json: still 0/25 resolved and 92% apply; infra_error 21 and test_fail 4, versus test_fail 23 and infra_error 2 before.

In the gold run, 10 of the 25 failures were test IDs the runner could not collect (8 Django, 2 SymPy). Thirteen more never ran a test in the host’s single Python 3.12 environment: missing packages, unbuilt C extensions, a module removed in 3.12, a pytest newer than the repository expected; six of those logs end in pytest’s “found no collectors”, with the cause cut off. Fixing the judge left the gold patch at 0/25, with 4 environment failures still booked as the model’s. We retired our own judge for the official Docker harness.

That day taught us nothing about capability. It taught us the thesis of this note.

The instrument lies more often than the model — and almost nobody audits the instrument.

The number we had to correct downward

We had been saying 10.1×. Then we measured it properly — same batch, three repetitions, the whole machine to ourselves — and it came out lower: 9.78× in the protocol-compliant window (σ 0.01 within that window). The same configuration measured 9.5–9.8× across sessions; an earlier window that did not follow the protocol put the baseline 9% faster. Two knobs moved: TP8 on 8 GPUs → TP2 + DFlash-64 on 2. Both windows decode a synthetic filler prompt greedily (a marker word plus one word repeated to 256 tokens, temperature 0, 512-token output cap); speculative acceptance depends on the text, so this is not a real-workload speedup.

TP8 no speculation 183.6 → TP2 + DFlash-64 1,795.1 tok/s (9.78×, window single10x-20260825-0035/resumo.json). In a later session on a shared node, TP2 alone ran at 112.7 tok/s and TP2 + DFlash-64 at 1,746.5 tok/s: speculation by itself is 15.5× (n=3 per arm, partp2-20260826-1950/resumo.json). Across sessions the configuration measured 9.5–9.8×.measured

Four replicas, conc=32 per replica: aggregate 29,503 ± 104 tok/s, n=3, overlap 118 s (composto4xc32-20260825-0244/resumo.json). The archive has no interference percentage.measured

The gain is 10.1× over baselineretracted

Solid ink means repeated and archived. Light means model prediction, not measured. Struck through means we published it and later took it back.

Why it moved

The 10.1× divided one 90-second reading of TP2 + DFlash-64 on 21 Aug (1,782 tok/s, 256-token prompts) by 176 tok/s, a TP8 baseline from 18 Aug measured with 8,192-token prompts. The comparable baseline, 256-token prompts on 20 Aug, read 187 tok/s; against it the same numerator gives 9.5×. The files of those windows are not in the public packages.

In the remeasurement, the first run gave 200.7 for the baseline. That reading is in single10x-20260825-0011/resumo.json: TP8 baseline 200.7 tok/s, n=3, σ 2.0, in a window without cron suspension; the DFlash arm was not measured (failed the content gate). It was retracted in favor of window 0035 (cron suspended, isolated node, both arms in the same batch, single10x-20260825-0035/resumo.json): 183.6 tok/s. We do not treat the cause of the 200.7 → 183.6 difference as proven.

In the protocol-compliant window (single10x-20260825-0035/resumo.json), TP8 no speculation at 183.6 → TP2 + DFlash-64 at 1,795.1 tok/s yielded 9.78× (σ 0.01 within window). In a later session on a shared node, TP2 alone ran at 112.7 tok/s and TP2 + DFlash-64 at 1,746.5 tok/s: speculation by itself is 15.5× (n=3 per arm, partp2-20260826-1950/resumo.json). Across sessions the configuration measured 9.5–9.8×.

That is the whole point. A bad measurement does not announce itself. It returns a plausible number, on time, with exit 0.