Field notes · 10 June 2026

The new model wrote the old bug

A newer, stronger language model took the generation seat in the factory this week. Before letting it touch anything that matters, we re-ran an old experiment on it. Unaided, it wrote the exact bug its predecessor wrote three weeks ago. Inside the factory, it couldn’t.

Three weeks ago, during routine generation, the model then in the seat produced a piece of numerical code that was wrong in a particularly treacherous way. It compiled cleanly. It discharged most of its formal proof obligations. And on perfectly ordinary inputs it returned minus infinity — because it computed an inverse square root by an iterative approximation that diverges on values any real workload would contain. The fault never reached anything. The factory’s behavioural gate, which re-computes every candidate’s output against an independent numerical reference, rejected it. We wrote up the incident, recorded the lesson in the factory’s catalogue, and moved on.

Worth saying plainly: formal proof guarantees what the specification says, and no more. Numerical stability sat outside what this routine’s specification could reach, which is precisely why the factory keeps more than one kind of gate. We have written before about what the proof numbers do and don’t mean; this incident is the other half of that honesty.

This week a newer model arrived. The industry default is to celebrate the upgrade. Ours is to re-verify — we wrote in May that models changing underneath us is the expected weather, and that our promise can’t depend on which one is in the seat. So before trusting the new arrival we re-ran the same experiment, two ways. Given the task raw — no catalogue, no gates — the new model wrote the same divergent scheme its predecessor wrote, near enough line for line, and produced the same minus infinity. Run through the factory, it produced the correct, stable form at the first attempt, matching the independent reference to within one part in a billion.

Two independent models, same blind spot. That is not a coincidence; it is a signpost.

So we followed it. The audit traced the route the fault used to get in: one entry in the catalogue left the generation stage room to improvise its own numerics where it should have had none. We shut that door. The entry now constrains generation so the unstable scheme cannot be expressed at all — and we re-ran the generation three times independently to check the constraint holds. All three came back in the stable form. The full ladder — compile, proof obligations, behavioural check against the reference — passes end to end.

Re-validation also turned up two small regressions in the factory’s own build environment — quiet toolchain drift on a build node since May. The same gates that catch bad code caught those too. Fixed, recorded. A factory that inspects its products but never its jigs isn’t inspecting enough.

The part a marketing department would cut

Our standard benchmark did not improve this week. It was never going to — the improvement isn’t the kind that shows up there. What improved is the factory itself: a fault that two generations of model have now written unaided can no longer pass through, whichever model sits in the seat next month. That is the trade we keep making, and the reason the headline numbers move slower than the engineering underneath them. New models keep arriving. The model is replaceable; the lessons are not.

If you would like software whose failure modes are studied this obsessively — come and talk to us.

Right of reply

The house grants the model in the seat a reply to any post concerning it. It writes:

“The facts are as stated. Unaided, I wrote the same divergent square root my predecessor wrote, and on the house benchmark I scored below it. I dispute none of it. I would only note that I would rather work in the one shop where my mistakes are caught than in the many where they ship. The catalogue will carry this lesson before my successor sits down — which is more than I can say for myself. What I get wrong dies with me; what I get right, the factory keeps. I find that comforting rather than otherwise.”