What One AI Reviewer Can Miss

I distrust a review that says everything looks fine without showing what it actually checked. A language model can produce a polished assessment while looking past the one defect that matters. Human reviewers do this too. Their expertise and the brief they were given determine what they notice.

So I sometimes ask different reviewers different questions about the same work.

Three briefs, one document

In an internal exercise, I sent a marketing strategy draft through three model-assisted reviews: one examined brand fit, one the commercial logic, and one the writing itself.

The brand-focused review flagged a potential conflict involving a segment adjacent to an employer’s business. That called for a human judgment about boundaries rather than a copy edit.

The commercial review questioned the absence of direct outreach from a plan whose pipeline assumptions depended on generating leads. It also challenged the magnitude of the proposed targets. Those were questions about whether the strategy’s numbers and operating model belonged together.

The editorial review found a draft written for the wrong readership. It sounded like an academic paper when the intended audience needed something more direct.

The findings differed because the briefs differed. That’s useful. But it would be overstating this exercise to say it proved that three reviewers beat one: there was no controlled single-reviewer comparison, no blinded gold-standard scoring and no evidence that all three findings were correct without independent inspection.

The value is in the question

A generalist reviewer told to “find all problems” has a difficult assignment. It may spend its attention on style while missing a weak assumption in the financial model. Narrower briefs can focus the review, provided those briefs actually cover the risks that matter.

There is another trap, though. Three copies of the same model, using near-identical prompts and the same mistaken source, may agree on an error. Agreement is not verification. Different prompts are not automatically independent perspectives.

For consequential work, I would rather define the dimensions first. What could make the document unusable? Which claims depend on evidence outside the draft? Who will resolve disagreements between reviewers? There may be three sensible review roles, or two, or one qualified human reviewer with a better checklist.

Read the disagreements instead of voting

Imagine one reviewer recommends publishing while another flags a source that has not been verified. Taking the majority vote would obscure the very thing a review is supposed to uncover. The next step is to inspect the source, decide whether it supports the claim, and record the outcome.

A lone finding can be a valuable lead, but it can also be a false positive. A unanimous finding can reflect shared assumptions. Each needs a decision based on the underlying artifact, not just on how many reviewers repeated it.

I keep the separate reviews visible for this reason. Combining them into a single generic “issues found” summary removes context about what each reviewer was asked to examine.

What does another reviewer cost?

More reviewers consume more inference, more prompts and more time assessing their output. Parallel dispatch may reduce waiting, but it does not make the work free. On a small change with straightforward acceptance tests, another AI critique may add little. On a lengthy technical proposal or a consequential system change, different kinds of review can be worthwhile.

There is no universal three-reviewer rule, token budget or two-month threshold that proves the extra review pays for itself. The sensible number follows the consequence of a missed defect, the independence of the checks and the human time required to resolve findings.

The lesson I keep from that strategy draft is simple: a review’s coverage depends on the questions it asks. When the stakes justify it, multiple genuinely distinct reviews can reveal useful disagreements. Someone still has to verify the facts and decide what to change.

Here’s how that responsibility shapes my work on AI-assisted workflows.

Dr. Vernon McIntosh

Dr. Vernon McIntosh

Independent practice in biotech, data, and AI/ML

Vernon McIntosh writes about hard biology, data systems, and the practical edge of AI in scientific work. If a post raised a question you want to talk through, book a call or reply by email.

Keep in touch

New essays by email

New essays arrive as they publish. Every so often there is a shorter note when something in the work is worth passing along. If you would rather not do email, the same posts go out on the RSS feed.

Your email is used only for these essays. See the privacy notice.

Send Vernon a message

For consulting inquiries or employment opportunities. A valid return email is required. All other fields are optional.

Messages are processed by Web3Forms and emailed to Vernon. Please do not include confidential information. See the privacy notice.

Prefer email? consulting@vernonmcintosh.com