How institutions measure and bound LLM hallucination in production — evaluation, runtime guardrails, monitoring, human sampling — and who helps.
A customer assistant that invents a clause in a mortgage product is not an IT bug. It is a conduct problem, a complaints generator, and — depending on what the customer did next — a remediation case. As language models have moved from pilots into production across European financial services, the question institutions ask has shifted accordingly: not whether models fabricate, which is settled, but how the fabrication rate is measured, bounded and evidenced while the system runs.
That question now has an operational discipline attached, and a vendor market forming around it. Both are worth understanding precisely — because monitoring is a control, not a cure, and buying it as a cure is how institutions end up with a dashboard and the same risk.
Hallucination in a financial institution has a specific shape. An assistant cites a policy document that does not contain the cited passage. A summarisation tool attributes a figure to the wrong counterparty. An eligibility answer states a threshold the product never had. Each output arrives fluent and confident; the failure is invisible at the moment of delivery, which is what distinguishes it from most operational risk.
Regulatory expectations have caught up on both sides of the Atlantic. The EU AI Act's accuracy and robustness requirements sit in Article 15 — deferred for high-risk systems to December 2027, but already the measuring stick in supervisory conversations. In April, the US banking agencies replaced their long-standing model risk guidance with a framework that explicitly extends model-risk expectations to generative AI, including customer-facing copilots. The direction of travel is identical everywhere: an institution running language models is expected to know its error rate, and to show the evidence.
The scale of the underlying problem is not hypothetical. The FinanceBench benchmark — built from questions answerable from public filings — found in its original study that a state-of-the-art model with retrieval failed 81% of its financial questions. Domain difficulty is the point: generic accuracy numbers travel poorly into financial documents.
The institutions that handle this well run four layers, and the vendor market maps onto them.
Pre-deployment evaluation. Before a system faces users, it is scored against a golden set — questions with known answers drawn from the institution's own documents, plus domain benchmarks of the FinanceBench type. Vendors such as Patronus AI and Braintrust operate here, supplying evaluation harnesses and open detection models. The institution's own golden set matters more than any public benchmark; it is also the artefact a model risk function can actually validate.
Runtime guardrails. An inline checking layer that scores each response for groundedness against the retrieved source material before the user sees it, flagging or blocking failures. Galileo and its peers advertise sub-200ms inline evaluation. The trade-off is real and worth stating: blocking aggressively protects customers and frustrates them in equal measure, every checker adds latency and cost, and the checker is itself a model with an error rate.
Production monitoring. Continuous sampling of live traffic, scored for faithfulness to sources — increasingly by evaluator models judging generator models — with rates tracked over time and fed into model-risk reporting. This is the layer that turns hallucination from an anecdote into a metric, and platforms of the Maxim AI class have built their offering around exactly this feed.
Human sampling. A QA regime that reads a stratified sample of real interactions, plus complaint-stream analysis as a lagging detector. Least fashionable, most trusted by regulators — the human-reviewed sample is what makes every automated number above it credible.
The monitoring stack measures the failure rate. Architecture determines it. Two decisions upstream of any monitoring tool do most of the work: grounding the system in retrieved documents rather than the model's memory — the RAG decision — and scoping the model to a bounded domain, where a smaller, purpose-built model has a smaller surface on which to invent. A model that only answers from the institution's policy corpus, and says so when it cannot, produces a hallucination-monitoring problem an order of magnitude more tractable than a general assistant with the same user base. This is the design territory Digiwit works in — purpose-built models that run inside the institution's perimeter — and the honest caveat applies here too: scoping reduces the rate; nothing yet published eliminates it.
If your institution is choosing between adding a monitoring layer and re-scoping the system underneath it — or suspects it needs both and wants the sequencing — that is a working session we do often.
Three limits survive every procurement in this category. Evaluator models hallucinate too, so a monitoring stack needs its own validation — the model risk function ends up watching the watcher, and should. Metrics without thresholds are decoration: a hallucination rate means something only once the institution has decided what rate triggers what action, in writing. And the accountability does not transfer — the named individual responsible for the system signs for its outputs whether or not a vendor's dashboard was green that week.
The market forming around hallucination monitoring is genuinely useful — the runtime and evaluation tooling is far better than what institutions could assemble two years ago. The institutions getting value from it share one habit: they treated the tooling as the measurement layer of a design decision they had already made, rather than as permission to skip the decision. Confidence in a system is earned twice — once in the architecture, once in the evidence. The vendors sell the second. The first is still yours.
Related reading:
Stop renting generic models. Start building specialized AI that runs on your infrastructure, knows your business, and stays under your control.