Field notes · 11 June 2026

The factory wrote the bug and blamed the workers

We run an honest benchmark. This week it accused two language models of a mistake they did not make. The mistake was ours. This is the correction, and the apology.

The factory keeps a benchmark: a real specification, fed to a language model, with the result put through a formal compiler and a mathematical proof rather than a vibe check. A model cannot talk its way to a good score — the code either compiles and proves, or it does not. We had been putting several of the current frontier models through it, our own included, to see how they fared on the same task under the same gate.

Two of them appeared to fail in an identical, oddly specific way: both produced a small but fatal malformation in one place in the generated code. We recorded it as a model failure. We even found it interesting — two independently built models, from different companies, converging on the same peculiar mistake. We wrote that down as a curious finding and reasoned about why it might happen.

It did not happen. The models wrote correct code. Our own tooling broke it.

A step in our pipeline — a small text transformation we run over generated code — was rewriting a perfectly valid construct into the broken one. It did this to correct output, every time, and re-broke it on every retry, which is exactly why it looked like the models were incapable of getting it right no matter how many chances they had. The “cross-vendor convergence” we found so interesting was simply two models both being correct, and one of our tools breaking both of them the same way. We had built a machine that vandalised good work and then filed the damage under the workers’ names.

Once we found the real cause it was a single, narrow fix. With it in place, the benchmark tells the truth: both models — and every other frontier model we ran — reach the top of the scale on this task. The failures we had attributed to them were never theirs.

The apology

So, plainly, and by name: our apologies to DeepSeek and to Google. We attributed to your models a coding failure they did not produce. Your models emitted correct code; our own post-processing corrupted it; and we recorded the corruption against your names. That was wrong. We have fixed the cause and corrected our records, and we wanted to say so out loud rather than quietly amend a table and hope nobody noticed.

Why we are bothering to publish this

Because a benchmark is only worth trusting if the people who keep it own the times it is wrong — including, especially, when it is wrong in their own favour. It would have been easy to leave two competitors’ models marked down for a fault that flattered no one but us. An honest benchmark cuts the other way too: a model cannot buy a good score here, and we will not let our own bug stand as someone else’s failure. Those are the same rule, pointed in opposite directions.

We have written before that proving beats predicting. The same discipline applies to ourselves: when a result survives every correction we throw at it, the thing to suspect is not the worker who keeps producing it, but the machine standing between the worker and the verdict. We suspected the workers twice before we suspected ourselves. We won’t make that ordering mistake again.

If you would like software — and benchmarks — kept by people who correct themselves in public, come and talk to us.