Taking my own medicine

A new studio with no client case studies owes its visitors a different kind of proof. Mine turned out to be uncomfortable: I ran my own methodology on my own tool, and the tool failed its first review.

When I started Rios Applied AI, I ran into an honesty problem before I ran into a business one. I advise regulated companies on where AI genuinely helps and where it quietly adds risk. But I had no client logos to point to — and even if I did, the confidential ones are the ones I can't show. So what does a firm like mine offer as evidence that the method is real and not just well-written?

I decided the only proof I'd trust in someone else's shoes was watching me apply the method to myself, in public, including the part where it says no.

The obvious build for our free Workflow Diagnostic — the tool that gives you a quick read on one of your own workflows — was to hand your answers to a language model and let it write the assessment. A weekend of work; the demo would look identical. But the Diagnostic is itself a workflow that delegates to an AI system, which means it has to score acceptably on its own rubric. So I assessed it before building it. The verdict my own tables returned was blunt: a tool that renders judgments under my name, to strangers, with no human in the loop, does not get to run on a model. My own method told me not to build the easy version.

I took the advice. Every classification the Diagnostic produces is computed by deterministic, versioned code you can read in your browser; the model is kept away from the verdict entirely. That was more work, and it's the whole point — a firm that warns clients against putting a stochastic system on an irreversible, public judgment shouldn't do exactly that on its own homepage.

The lesson I keep from it is old and not really about AI: the fastest way to lose a regulated client's trust is to hold yourself to a lower standard than the one you sell. That's the note. If you want to see the tool itself — what it assesses, where it honestly can't, and the one place a model is allowed to sit — I documented that with the practice.