Six AI-built features. Six full test suites, all green, zero failures. None of them worked once the data was saved and reloaded. I wrote about the judgment side of that story elsewhere: knowing when to reject a plausible answer is the actual skill now, not producing the first draft. This post is about the mechanics. What does a check that actually catches the problem look like, and how do you build it into a workflow instead of hoping you remember to look?
Plausible is not the same claim as correct
A model answer that reads well is doing exactly what it was trained to do. Fluency is not a byproduct of correctness, it is a separate skill the model has mastered independently. Same with code: a function that passes the tests the model wrote for it, in the same session, off the same assumptions, has passed a test that was never designed to catch the model’s own blind spot. If the model misunderstood the requirement, its test suite inherited the same misunderstanding. The suite going green just confirms the model agrees with itself.
In the six-feature case, every test exercised an in-memory copy of the state or called a service function directly. None touched the actual save-and-load path the application uses in production. The tests were honest about what they tested. What they tested was not the thing that mattered.
The verification ladder
I use four checks, in order, and I do not skip a level because the one before it passed.
1. Does it run. Compiles, executes, returns without an exception. This catches almost nothing on its own, but it is a five-second filter and there is no reason to skip it.
2. Does it pass a test the model didn’t write. If the model authored both the code and the test, the test is the model checking its own work against its own assumptions, not independent verification. A test written by someone else, or against a spec that predates the code, can fail in a way the implementer’s own test never would, because it was not built to confirm the same beliefs.
3. Does it reconcile against a known-good reference. Most teams skip this step because it takes real effort to set up, and it is the step that matters most. A financial model has to tie out to the general ledger. A pipeline output has to match a hand-computed result on a small, known input. A generated report has to trace every number back to a source document, not to a plausible-sounding figure the model produced. If there is no ground truth to check against, that is worth noticing on its own, because it usually means the check being proposed is not actually a check.
4. Does a human who owns the outcome sign off. Not a human who reviewed the diff. A human whose name is on the result if it is wrong. Review is cheap when the reviewer has no stake in the answer. Sign-off from the person accountable for the outcome is a different kind of scrutiny, because they are the one who has to explain it later if it breaks.
The ladder only works if you do not treat step one as if it were step four. A passing build is not a passing reconciliation, and a code review is not a sign-off. Conflating them is how six green test suites become six broken features.
The seam is where it breaks
Every failure I have caught in this category broke at a seam, not inside a single function: the save-and-load round trip, the handoff between the pipeline stage that generates a number and the report that cites it, the point where a value crosses from the model’s context into a document someone else reads without re-deriving it. Unit-level correctness inside one function tells you almost nothing about whether the seams hold, and seams are exactly where an AI-generated implementation is most likely to be wrong, because the model wrote each piece in isolation and never had to reason about how they connect in production.
Put your verification budget at the seams first. A function that is locally correct but wired into the wrong seam produces a result that looks fine and is not, right up until someone uses it the way the application actually uses it.
Build the check into the workflow, not the memory
The reason all six features shipped broken is not that nobody was smart enough to catch it. The check was not built into the process, so it depended on someone remembering to run it. That does not scale, and it fails exactly when you are busiest, which is when AI output is most tempting to trust without question.
The fix is structural. Put the reconciliation step in the merge checklist instead of trusting someone to remember it under deadline pressure, and put the source trace in a template field instead of relying on habit. The sign-off should be a name on the document, not an implicit assumption that someone looked at it. Trust in AI output has to be earned per run, against a specific check, not extended by default because the last five runs were fine.
The model will not tell you when it did the wrong thing correctly. It cannot see that failure mode in itself. The workflow around it has to be the thing that can.
If you want a second pair of eyes on where your team’s AI output actually gets checked before it ships, book a scoping call and I will walk through where the gaps usually are.