Your AI Is in Production. When Should You Turn It Off?

Most organisations now have a process for getting AI into production: a business case, a pilot, a sign-off. Far fewer have a process for getting it out. Once a system is live, it gets a budget line, an integration, a team that depends on it and a story about why it was a good idea. From then on, the question is usually how to improve it, not whether to keep it.

That is a gap worth closing, and the reason is not that AI tends to fail. Any live system can drift away from the reason it was approved. Its costs change, the quality of its output changes, and the work around it changes. If nobody has agreed in advance what “no longer worth running” looks like, that judgement is made late, under pressure, or never.

What the survey says, and what it does not

On 19 October 2026 Gartner published results from its survey of 2,002 CIOs and technology executives, conducted between May and July 2026. Three figures matter here:

• 13% of respondents said their enterprise had realised significant or transformational value from AI.
• 31% said they had guidelines for when pilots should not move into production.
• 17% said they had guidelines for abandoning low-value initiatives that are already in production.

Gartner also reports that 51% expect AI to raise their technology total cost of ownership. Organisations with pilot stop rules were 2.3 times more likely to belong to its “vanguard” group; those with production sunset rules were 2.6 times more likely. Gartner defines vanguards by better business-outcome performance alongside high reported CEO-perceived AI value.

These numbers need careful reading. They are self-reported answers from technology leaders, not measured outcomes. The vanguard comparisons are associations within Gartner’s own classification. They do not show that writing a stop rule causes better results. Organisations that are disciplined enough to write sunset rules may simply be more disciplined in general. None of these figures describes Boesing clients or applies as a rate to any particular organisation.

The useful signal is narrower: in this sample, production sunset rules were rarer than pilot stop rules. A separate Gartner release, published 20 October 2026, reports that 44% of CEOs struggle to demonstrate AI-driven financial returns to stakeholders. Its 2027 CEO Survey covered 203 global CEOs and senior business executives in April–May 2026. This is self-report, not proof of losses or absent value. The release does not specify the 44% question’s respondent base separately, and the two surveys cannot be combined into a failure rate.

Why live systems are harder to stop than pilots

A pilot is designed to end. It has a budget, a timeframe and a question to answer. A production system has none of those by default. Three things make it sticky:

• Integration. Other processes start relying on its output. Turning it off touches every workflow downstream.
• Attribution. Once a system is embedded, it becomes hard to say what it contributes. Results improve or decline for many reasons at once.
• Sunk cost. The money already spent on licences, integration and training feels like a reason to continue. It is not. That money is gone whether the system runs for another year or is retired tomorrow. The only costs that belong in the decision are the ones still ahead: running costs, oversight effort, rework, risk exposure and the value of the alternative.

Ask: knowing what we now know, is continuing better than the alternatives? Compare remaining costs and benefits, including transition costs, and remember that continuing ties up budget, staff and attention that could support other work. Count released capacity as a cash saving only where spending actually falls.

What drift looks like in practice

The question is not “is the AI good?” It is “is the AI still doing the job it was approved to do, at a cost and risk we would still accept?” Four kinds of drift are worth watching:

• Cost drift. Usage-based pricing, model upgrades, added integrations or growing human review can move running costs well beyond the original case.
• Quality drift. Output accuracy or usefulness can decline as the inputs, products, policies or customers change around the system.
• Safety and compliance drift. New uses, new data or new regulation can move a system into a risk category it was never assessed for.
• Value drift. The problem it solved may have shrunk, been solved another way, or never been as large as assumed.

None of these needs sophisticated measurement to spot. All need someone expected to look.

Public guidance supports the principle, not specific thresholds. The NIST AI Risk Management Framework 1.0 (26 January 2023) is voluntary, not law. In its Core, GOVERN 1.7 calls for processes for phasing out AI systems safely, without increasing risk or reducing trustworthiness; MANAGE 4.1 calls for post-deployment monitoring plans covering user input, appeal and override, decommissioning, incident response, recovery and change management. The NIST Manage playbook adds voluntary guidance, and NIST AI 800-4 (announced 9 March 2026, published 6 March) maps open monitoring challenges. None of this proves a return on intervention.

For systems classified as high-risk under the EU AI Act, there are specific post-market monitoring duties for providers (Article 72) and monitoring duties for deployers (Article 26). Those duties apply to high-risk systems, not to every AI tool. Whether a given system falls into that category is a legal question this article does not answer.

A three-way decision, not a yes or no

Framing the review as “keep or kill” pushes people to defend the system. A three-way choice is more honest:

• Keep. It still does the approved job at an acceptable cost and risk. Confirm the owner and set the next review date.
• Rework. The purpose is still valid, but the scope, model, process, controls or cost need to change. Set a fixed rework window and a clear test. If it doesn’t pass, the next decision is retire.
• Retire. The prospective costs and risks outweigh the prospective value, or a simpler alternative does the job. Plan the exit: who takes over the work, how outputs are archived, and how dependent processes are switched.

An operational sunset scorecard

The example triggers below are illustrative, not data or benchmarks. Each organisation should set its own before the review, not during it.

• Cost: current monthly running and oversight cost against the approved case. Example trigger: more than 25% above the approved figure for two consecutive months.
• Quality: error, correction or override rate against the level accepted at approval. Example trigger: human corrections double from the approved baseline.
• Safety and compliance: any new use, data type or regulatory classification since approval. Example trigger: any change not assessed at approval.
• Value: use, time saved or the outcome the system was approved for. Example trigger: usage falls below half the level assumed in the case.
• Alternatives: whether a simpler tool, process change or vendor now does the job. Example trigger: a viable alternative is available at materially lower prospective cost.
• Rollback readiness: whether the work can continue manually or elsewhere if the system stops. Example trigger: no tested fallback exists.

A trigger does not mean retire. It means the review happens now rather than at the next budget cycle.

One named owner and a 30/60/90-day rhythm

Sunset criteria need a named decision owner, even where a steering group approves the decision. That person should be accountable for the business outcome and authorised to order a rollback, with the builder and vendor providing evidence rather than controlling the decision alone.

A practical cadence after go-live or a significant change, offered as an example rather than a standard:

• 30 days. Confirm the system is used as intended, costs match expectations and the fallback works.
• 60 days. Compare cost, quality and value against the scorecard. Note any drift and who is investigating it.
• 90 days. Make an explicit keep, rework or retire decision and record it. Then set the next review interval, for example every six months.

The owner should also hold the rollback plan: what happens to the work if the system is switched off on Monday, and who has tested that.

The case against retiring too early

Sunset rules can also go wrong. Some AI systems need time: users learn, processes adjust and data quality improves. A system that looks weak at 60 days may be sound at 180. Retiring too early can waste the learning curve, damage trust in the next initiative, and push teams back to manual work that was worse.

Value is also unevenly visible. A system that prevents occasional errors may show little in a monthly dashboard and still be worth keeping. And frequent stop decisions can teach teams to avoid ambitious work.

Where value remains plausible but evidence is incomplete, use bounded rework or controlled observation: name the missing evidence, cap additional cost and exposure, and set a decision date. Do not extend indefinitely. An unresolved serious safety issue can require immediate suspension under the organisation’s incident plan; a cost trigger alone calls for review, not automatic shutdown. Match review windows to maturity and include hard-to-measure value explicitly. The goal is a deliberate decision, not a quick one.

Stop criteria start before go-live: From AI Pilot to Proven Business Value.

Where to start

If you have AI already running, choose one system and answer four questions this week. Who owns the decision to switch it off? What was it approved to do? What would make you review it now? Could the work continue tomorrow without it?

If those answers are unclear, the free Workflow Fit Diagnostic is a structured way to see where the workflow, ownership and dependencies stand before deciding what, if anything, needs to change. — Ronny Boesing