AI pilots fail at deployment rather than at proof. A pilot is built to remove the conditions that make production difficult, including messy data, absent ownership, undefined governance and ongoing maintenance. Success under those removed conditions predicts very little. The programme then fails on precisely what the pilot was designed to exclude.
A Pilot Is a Controlled Exclusion
The purpose of a pilot is to answer one narrow question. Can this technique produce a useful result on this kind of problem at all.
Answering that question quickly requires removing everything that would slow it down. Scope is narrowed, data is cleaned by hand, edge cases are set aside, and the best available people are assigned.
All of that is correct practice. A pilot loaded with production constraints would take as long as production and would prove nothing sooner.
The error is in the interpretation, not the design. A successful pilot is read as evidence that the capability works, when it is evidence that the capability can work under conditions nobody intends to maintain.
Feasibility and viability are separate questions. The pilot answers the first one honestly and says nothing at all about the second.
Programmes stall because the second question is never asked as a question. It is assumed to have been answered by the first.
The pattern repeats across technologies. Automation projects, analytics platforms and workflow systems have all produced convincing demonstrations that did not survive the move into daily operation.
What is different now is the speed of the demonstration. A capable result can be produced in days, which shortens the gap between an impressive demonstration and a funding decision.
What the Pilot Quietly Removed
The exclusions are worth naming individually, because each one becomes a failure mode later.
Data condition is the first. Pilot data is usually selected, cleaned and often corrected by hand, while production data arrives incomplete, inconsistent and formatted by whoever entered it.
Staffing is the second. Pilots are staffed by capable and interested people who volunteered, and production is staffed by whoever holds the role that week.
Attention is the third. A pilot runs with a sponsor watching, which resolves blockers in hours that would otherwise take weeks through normal channels.
Scope is the fourth. Pilots select the cases where the technique is likely to work and exclude the difficult remainder, which is often the majority of real volume.
Time is the fifth and least visible. A pilot runs for a fixed short period, during which nothing in the surrounding business changes enough to break it.
Each exclusion is defensible on its own. Together they describe an environment that has almost nothing in common with the one the tool will actually live in.
Integration is the sixth exclusion and often the largest. A pilot can accept input by hand and produce output into a spreadsheet, while production has to read from and write to systems that were not designed for it.
Deployment Tests Ownership Before It Tests Accuracy
The first thing production asks is who owns this. That question has no answer in most pilots, because the pilot team owned it by default.
Ownership means several distinct things at once. Someone decides when the output is wrong, someone fixes it, someone approves changes, and someone answers when a customer is affected.
In a pilot these roles collapse into the project team. In production they belong to different functions with different priorities and different reporting lines.
The gap shows up as delay rather than as failure. Output degrades, several people notice, and nobody has the standing or the time to act on it.
Unowned systems do not stop working suddenly. They drift, produce results that staff learn to distrust, and are quietly worked around until they are abandoned.
A clear structured route from a working pilot to a supported deployment exists precisely to force these assignments before the tool reaches live use.
Naming an owner is cheap at the start and expensive later. After launch, the assignment becomes a negotiation between functions that have already absorbed other commitments.
A useful test is to ask who would be interrupted if the output were wrong on a Friday afternoon. If the answer is the project team, the deployment is not ready.
The second test is to ask who has authority to switch it off. Systems without a named person able to suspend them tend to keep running through problems that should have stopped them.
Governance Is the Second Test
Governance sounds like a compliance concern and is actually an operating one. It determines what happens in the cases the pilot never encountered.
Every production system generates situations outside its tested range. A customer request phrased unusually, a document in an unexpected format, a case that sits between two categories.
Governance is the set of standing answers to those situations. What the tool is permitted to decide alone, what requires review, and what must be escalated to a person.
Without those answers, staff invent their own. Some accept the output uncritically, others check everything manually, and the variation between them becomes invisible until something goes wrong.
The absence is hardest to see when the tool performs well. Consistent good output removes the pressure to define boundaries, so the boundaries get defined during the first incident instead.
Organisations that move from experiment to routine use tend to follow a deliberate path from isolated pilots to standing governance rather than treating each deployment as a separate exercise.
Governance also settles the record-keeping question. What was decided, on what input, and by which version of the system, all of which become urgent the first time an outcome is disputed.
Staff behaviour is the other governance variable. People adjust their own work around the tool, and those adjustments become the real process regardless of what the documentation says.
Maintenance Is the Test Nobody Budgets
A pilot has a defined end date. A production system has an indefinite one, and that difference changes the economics completely.
Inputs shift over time. Customers phrase requests differently, suppliers change formats, regulations move, and the business itself changes what it sells.
Performance degrades gradually against all of that. The decline is slow enough that no single week justifies attention and cumulative enough to matter within a year.
Maintenance means monitoring output quality, refreshing reference material, adjusting prompts or rules, and retiring behaviour that no longer fits. None of that appears in a pilot budget.
Vendors rarely raise it either, because the maintenance burden falls on the buyer and discussing it complicates the purchase. The subscription is quoted. The standing internal effort is not.
The practical consequence is a slow reversal. Staff notice output quality falling, revert to the previous method for the important cases, and the tool survives only for work that does not matter much.
Version changes add a further layer. Underlying models and platforms are updated by vendors on their own schedule, and behaviour can shift without anything changing on the buyer side.
That makes periodic revalidation part of the standing cost. A short recurring check against a fixed set of known cases catches most of this before users do.
Designing a Pilot That Predicts Deployment
A pilot can be made more predictive without losing its speed. The adjustments are small and they change what the result means.
Use unedited data for part of the test. Cleaned data proves the technique works and dirty data proves whether the workflow around it survives contact with reality.
Include the difficult cases deliberately. A pilot that handles only the straightforward majority hides the exact volume of exception work that production will generate.
Assign the eventual owner during the pilot rather than after it. That person surfaces objections early, and those objections are the cheapest information the pilot produces.
Run at least part of the pilot without sponsor attention. How the process behaves when nobody senior is watching is the closest available preview of normal operation.
Write the governance rules before launch rather than during the first incident. Three or four written boundaries covering permitted decisions, required review and escalation are usually enough to start.
Set the success criterion at the process level, not the model level. Accuracy on a test set is a technical result, while cycle time and exception volume are business results.
The uncomfortable implication is that a pilot which fails under these conditions has done its job. It has produced the information the organisation needed at the cheapest possible moment. A pilot that succeeds under favourable conditions and is then scaled has produced confidence instead of information, and confidence is what makes the programme expensive. The question worth asking after any successful pilot is not whether the technology worked. It is which of the removed conditions the business is now prepared to put back.
None of these adjustments require a longer pilot. They require the pilot to answer a slightly different question, which costs planning attention rather than calendar time.
The choice is between paying for that information early or paying for it after the programme has been announced internally. The second option carries reputational cost alongside the financial one.
Frequently Asked Questions
What is the difference between a pilot failing and a deployment failing?
A pilot fails when the technique cannot produce a useful result, which is a technical outcome and usually cheap. A deployment fails when the technique works but the organisation cannot sustain it, which is an operating outcome and usually expensive. The second failure happens after budget, staffing and expectations have all been committed. Most programmes that stall have already passed the technical test.
Should a pilot use real production data?
At least part of it should. Cleaned data answers whether the technique is capable, while unedited data answers whether the surrounding workflow can cope with what arrives in practice. Running both gives two different and equally useful results. Using only cleaned data produces a pilot that cannot fail in the way that matters.
Who should own an AI tool once it is live?
Ownership belongs with the function that owns the underlying process rather than with the technology team. The technology team maintains the system, but the process owner decides what acceptable output looks like and when it has stopped being acceptable. Splitting these roles without naming both leaves the monitoring gap that most deployments fall into. The assignment should be made before launch.
How much maintenance does a deployed tool actually require?
Enough that it needs a named person and recurring time rather than occasional attention. The work includes checking output quality, updating reference material, adjusting rules as the business changes and retiring behaviour that no longer fits. It is ongoing rather than periodic, because the inputs shift continuously. Treating it as a one-off cost is the most common budgeting error.
What governance rules matter most at the start?
Three questions cover most of the ground at launch. What the tool may decide without review, what requires a person to check before it takes effect, and what must be escalated regardless of confidence. Writing those answers down before launch prevents each staff member from inventing a private version. The rules can be refined once real cases start arriving.
Is it worth running a pilot at all if the conditions are artificial?
Yes, provided the result is read correctly. A pilot is a cheap answer to a narrow question and remains the right first step for most organisations. The failure lies in treating that narrow answer as permission to scale rather than as one input among several. Adding a few production conditions to the pilot preserves the speed while improving what the result actually tells anyone.
No comments:
Post a Comment