What a benchmark is really a contract for

Early in my career I led the rollout of a new collateral-management system — trained the users, stood up the audit trails, and owned the answer when someone asked whether the thing could be trusted. That experience is why I read the AI industry's new benchmarks as a procurement story before a research one.

When you're the person implementing a vendor's system inside a regulated shop, you learn to distrust the demo. The demo is the vendor's best case, run on the vendor's setup, by people who built it. What you actually have to certify is different: will this hold up on our data, in our process, when an examiner pulls a sample twelve months from now? The gap between those two things is where careers get made or ended.

So when I watched every frontier lab quietly retire the old knowledge benchmarks — the trivia-style tests — and start leading with computer-use scores, tasks run against real software and browsers and terminals, I didn't read it as a leaderboard update. I read it as the industry changing the contract. A benchmark is a formal statement of what "good" means. Change the benchmark and you've changed what buyers are supposed to be paying for.

And here's the part my old job made obvious. These new scores are a property of the whole system — the model plus the scaffolding around it — not of the model you actually license through an API. Two teams can take the same model and score ten points apart depending on their harness. Which means the vendor's headline number is, once again, a demo. The right procurement question isn't "what did you score?" It's "what harness produced that score, and how far is it from the configuration you're actually selling me?"

That's an unglamorous question, and it's the whole of this note: distrust the demo, and make the vendor show you the harness that produced the score. It's the question I'd have asked from the buyer's chair, and the one most keynote slides are built to keep you from asking. Turning it into an actual procurement discipline — what to measure, where the numbers mislead, what belongs in the contract — took a longer brief, which I wrote up for my practice.