The contract I watched break

For most of my career, one rule was so basic nobody said it out loud: the same input produces the same output. Then I started building with language models, and watched that rule stop holding — which told me exactly where the trouble was going to land.

I spent years in the parts of a bank where reproducibility isn't a preference, it's the job. Collateral controls under Dodd-Frank and ISDA. Cross-asset risk reporting. Reconciliations that had to tie out to the penny and survive an examiner asking, months later, "show me how you got this number." Every one of those processes rested on the same quiet assumption: run it again and you get the same answer. When something was wrong, you could replay it, find the broken step, and fix it. The defect stayed put.

That assumption is invisible until it's gone. And the first serious thing I noticed, building with large language models, is that they don't honor it. Same prompt, different answer. Reword the question harmlessly, get materially different behavior. The failure doesn't sit still long enough to bisect. That isn't a bug you patch — it's the nature of the tool.

The people debating which model is best were, to me, asking the wrong question. I'd sat on the other side of enough audits to know the durable question wasn't which model. It was: given a tool that won't reproduce itself, where in the system do you still require reproducibility — and where can you afford not to? Draw that line in the wrong place and you've put a statistical process where an examiner expects a guarantee.

Banks already solved a version of this, long before AI. It's why a wire transfer has maker-checker controls even when everyone involved is competent. Nobody sane sits an unmediated judgment call on an irreversible, high-consequence action. The instinct I'd built over fourteen years turned out to be the right instinct for this problem — it just needed to be made explicit and put on a diagram.

That's the observation, and it stands on its own: the oldest contract in enterprise computing has quietly been broken, and the people who spent careers living by that contract are the ones best placed to see where it now has to be enforced by design rather than assumed. Where exactly to draw that line — the compounding arithmetic behind it, where workflows end and agents begin — is a longer argument. I worked it into a full framework for my practice, for anyone who wants the systematic version.