Most AI projects in financial services fail at the transition from pilot to production. Six gaps that cause failure and how to close them.
The pilot was a success. The model performed well on test data, the stakeholders were impressed, and the business case looked compelling. Then it went into production, and everything changed. Accuracy dropped. Costs escalated. Latency became a problem. Compliance raised concerns that nobody had considered during the experiment. Six months later, the project was quietly shelved.
This is not a rare outcome. AI projects in financial services often fail at the transition to production rather than at the pilot stage — because the conditions that made the pilot succeed are precisely the conditions that do not exist in a real operating environment.
This article identifies the specific gaps between pilot and production that cause AI projects to fail in financial services, and what to do about each one before the transition begins.
A pilot is a controlled experiment. Production is an uncontrolled environment. The gap between them is not a single problem — it is a set of overlapping differences that compound each other.
Pilot datasets are curated. They are selected to represent the use case, cleaned of anomalies, and sized to be manageable. Production data is none of these things. It contains edge cases the pilot dataset did not include — unusual transaction patterns, incomplete customer records, documents in unexpected formats, multilingual inputs the model was not tested on.
In engagements we have reviewed, a model that scored above 90% on a pilot dataset has dropped into the high 70s on production data — not because the model degraded, but because the real world contains a distribution of inputs that the pilot did not represent. The accuracy number that justified the business case was measured under conditions that no longer apply.
A pilot might process a few hundred queries per day. Production processes tens of thousands. This changes the economics: API costs that looked manageable at pilot scale become significant at production volume. It changes the infrastructure requirements: a model that responded in 500ms during the pilot may queue requests and respond in 3 seconds under production load. And it changes the failure profile: at low volume, a 5% error rate means a handful of incorrect responses per day that a human can catch. At high volume, it means hundreds of errors per day that overwhelm the review process.
During a pilot, every AI output is reviewed before any action is taken — because the pilot is itself a review exercise. In production, the human review architecture has to be designed deliberately rather than assumed. For many regulated processes — credit decisioning, AML alert disposition, customer suitability assessments — human oversight is not optional; the EU AI Act's high-risk obligations and sectoral rules such as MiFID II and consumer credit regulations explicitly require it. The question is not whether humans review AI outputs but where they sit in the workflow, and what the AI layer is doing to make their review effective rather than performative. A hallucinated regulatory citation that was caught during pilot review becomes a compliance violation if the production design omits the equivalent checkpoint.
Pilots often operate outside the formal compliance perimeter. They are experiments, proofs of concept, sandbox exercises. Production deployments are operational systems subject to the full weight of regulatory requirements — EU AI Act high-risk obligations, DORA third-party oversight, audit trail requirements, data governance standards. Many pilot projects are approved without compliance involvement and only encounter regulatory scrutiny when they are proposed for production. By that point, the architecture may not support the controls that compliance requires.
The single most effective step is building an evaluation dataset that reflects the full distribution of inputs the model will encounter in production — including edge cases, incomplete data, multilingual inputs, adversarial examples, and the specific failure modes that matter most for your use case.
This evaluation dataset should be larger than the pilot test set (200-500 examples minimum), include cases the model is expected to get wrong (to measure failure behaviour, not just success rates), and be validated by domain experts who understand the real-world complexity.
If the model's accuracy on this dataset is materially different from its pilot accuracy, you know the gap before production — not after.
Run the system at production-scale query volume before committing to a production launch. Measure throughput, latency under load, queue depth, and failure behaviour when the system is saturated. If you are using an API, test against the provider's rate limits and measure what happens when those limits are hit during a peak period.
If you are running on your own infrastructure, this is where the hardware sizing decision gets validated. A single GPU that handles pilot volume comfortably may need a second GPU, a larger model server, or a load balancing layer to handle production throughput. Discovering this during a load test costs hours. Discovering it in production costs credibility.
Involve compliance from the beginning of the production planning process — not as a final approval gate. The EU AI Act's high-risk requirements demand risk management documentation, data governance, transparency logging, and human oversight. These are architectural requirements, not paperwork exercises. They need to be designed into the system, not bolted on afterwards.
For institutions using third-party AI APIs, DORA's third-party oversight requirements add another layer: contractual governance, audit rights, resilience testing, and exit strategies. These take weeks or months to negotiate. Starting this process after the production launch date is set creates deadline pressure that leads to compromises.
A pilot runs for a defined period and produces a final accuracy number. Production runs indefinitely, and accuracy changes over time. Customer behaviour shifts. Regulatory requirements evolve. New transaction patterns emerge. The model's training data becomes progressively less representative of the current environment.
Production systems need continuous monitoring: accuracy tracking against a rolling evaluation set, drift detection that flags when the distribution of inputs changes significantly, output quality sampling where human reviewers regularly assess a random sample of AI responses, and alerting thresholds that trigger intervention when key metrics deteriorate.
Without this monitoring layer, the institution has no way of knowing whether the system that performed well at launch is still performing well six months later. By the time someone notices a problem, the damage — compliance errors, incorrect decisions, degraded customer experience — has already accumulated.
Model accuracy degrades over time as the world changes and the model's training data becomes stale. Regulations are updated. New product types are introduced. Customer demographics shift. The model that was fine-tuned on last year's data is progressively less accurate on this year's inputs.
Institutions using open-weight models on their own infrastructure can retrain on their own schedule — quarterly is a common cadence for financial services applications. This requires maintaining the fine-tuning pipeline, keeping evaluation datasets current, and having a validation process that confirms the retrained model performs at least as well as the previous version before it goes into production.
Institutions using proprietary APIs do not control when the provider updates the model. A provider update can change the system's behaviour — for better or worse — without the institution's knowledge or consent. This is a production risk that does not exist during a pilot, because pilots are short-lived enough that provider updates rarely occur.
Human-in-the-loop is the norm for financial services AI deployments — and in regulatory-heavy processes such as credit decisioning, AML alert disposition, sanctions screening, and high-risk advisory, it is mandatory. The EU AI Act explicitly requires human oversight for high-risk AI; sectoral rules add further obligations. The production design question is not whether to keep humans in the loop, but how to position them and how to make their review meaningful at scale.
The most effective pattern in regulated contexts is structured review: the AI processes every case and produces a recommendation, a confidence score, and a reasoned narrative; a human reviewer accepts, modifies, or rejects it; the system logs the full chain for audit. Lower-risk processes can use sampling — humans assess a representative slice of AI outputs and triage exceptions — but the choice between structured review and sampling is a regulatory and risk decision, not just an efficiency one. The thresholds and architecture need to be calibrated on production data, not pilot data, because the distribution of routine versus complex cases may be different from what the pilot suggested.
This is also where analyst fatigue becomes relevant. If the human review queue is too large, reviewers rush through cases and the review becomes performative rather than substantive — a dynamic familiar to anyone who has worked through AML alert backlogs at scale. The architecture needs to ensure that the volume reaching human reviewers is manageable enough for genuine scrutiny, which means the AI layer needs to be accurate enough to handle the bulk of routine cases without escalation.
The gap between pilot and production is not primarily a technology problem. It is a planning problem. The pilot proves that the technology can work. Production proves that the organisation can operate the technology at scale, under real-world conditions, within its regulatory obligations, and with sustainable economics.
The institutions that succeed at the pilot-to-production transition are the ones that plan for production from the beginning — choosing the right base model for production constraints rather than pilot convenience, designing the compliance layer as part of the architecture rather than as an afterthought, and building monitoring and retraining into the operational plan rather than hoping the launch-day performance will persist indefinitely.
The pilot should not be an experiment that proves AI works. It should be a rehearsal for the production system that will follow.
If your institution has a successful AI pilot and is planning the transition to production, we can help you identify and close the gaps before they become problems.
Related reading:
Stop renting generic models. Start building specialized AI that runs on your infrastructure, knows your business, and stays under your control.