A research lead at a biotech firm told me recently that they had pulled ChatGPT Enterprise out of a literature-review workflow after it invented three citations in one internal memo. The team’s conclusion was simple: the model was not reliable enough.
My question back was also simple. What did you actually feed it?
There was a long pause. They had pasted a question into a chat window with no source documents attached, no retrieval path in front of it, and no verification step behind it. Of course it made things up. That was not only a hallucination problem. It was a governance problem wearing a model-problem label.
I keep having versions of this conversation. A team tries a capable model on an unstructured task, gets a confident wrong answer, and decides the technology is not ready. The model gets blamed for what the surrounding workflow failed to provide.
The number that should change the conversation
The HUMBR paper from researchers at Meta and University College London gives a result I now bring into scoping calls. In their legal-domain evaluation, the model with a structured consensus layer contradicted the source text 2.5% of the time, compared with 4.7% for human-drafted text, and cut missing key sections from 28% to 0.8%. The model did not change between the two arms. The layer around it did.
That gap is the point. When leaders treat hallucination as if it were mainly a property of the model, they reach for fixes that do not help much: wait for the next release, swap vendors, restrict access, or tell staff to be more careful. When they treat hallucination as a context and governance failure, the fix becomes engineering.
You ground the model in the right sources. You retrieve the right chunks. You add a check that catches residual errors before they reach someone who trusts the output. None of that requires a mythical model that never makes a mistake. It requires a pipeline.
I have seen the pattern across biotech, civil engineering, and federal R&D work. The teams getting useful production output from LLMs are not always the teams with the newest model. They are the teams with the cleanest context and a check behind it.
What governed actually means
Governed data is not a vague call for better hygiene. In the workflows I am willing to stand behind, it means three practical things.
First, the source set is defined. Not whatever the model remembers from training, and not whatever someone happened to paste in. A specific corpus, a specific version, and a specific retrieval path. If the model is summarizing literature, it is summarizing a known set of PDFs, not the open internet.
Second, retrieval is grounded. The model gets the right chunks for the question, with citations attached, before it generates anything. RAG done well is not decoration. It is the mechanism that turns a confident guess into a grounded answer, and it is the gap between the human-level contradiction rate and the HUMBR rate in the paper.
Third, the output is checked. A second pass, either rule-based or model-assisted, verifies that every relied-upon claim traces back to the retrieved source. Anything that cannot be traced gets flagged or refused, not shipped. This is the same cited-or-absent discipline I build into provenance-clean workflows. The check is what turns a lower error rate into something a regulated or liability-sensitive firm can use.
The tooling is not the bottleneck
The Braintrust landscape review walks through the detection-tool field, and technical leaders should notice that the tooling has matured past the point where detection is the hard part. Braintrust, Galileo, Arize Phoenix, Patronus AI, and Promptfoo cover stages such as pre-deployment evals, runtime guardrails, self-hosted observability, regulated-domain evaluators, and CI-native red-team probes.
A team that wants to know when factuality drops after a prompt rewrite can measure it. The infrastructure exists. What often does not exist is the decision to wire it into the workflow, define the fail condition, and name the person accountable for acting on what the check surfaces.
Detection without governance is a dashboard nobody looks at. Governance without detection is a policy nobody enforces. You need both, and you need one owner whose name is on the loop.
Stop waiting for the model to fix this
The teams I work with that ship AI-assisted output into real deliverables did not wait for a model that never hallucinates. They stopped treating the model as the source of truth and started treating the context pipeline as the source of truth. Inside that boundary, the model is a fast, fallible assembler.
That distinction matters. The model is allowed to be wrong during generation. The pipeline is not allowed to pass the wrong thing through as if it were checked work.
If your team is stuck on “the model is not reliable enough,” the question worth asking is whether you have built the pipeline or whether you are still opening a chat window and hoping. If you want a second pair of eyes on where the governance gap is hiding, the AI Workflow Diligence Sprint is a fixed-fee scoping call built for exactly this.