Where AI fails

The honest failure catalogue, from hallucination and drift to data debt and the pilot-to-production gap, and the rules that keep each of them from becoming your problem.

Listen to this lesson · narrated by Laurence Liew

Four failure cards in a grid: confident fabrication, drift, data debt, and the demo that never ships

Quick win this week. Check the last AI output you accepted without reading. Notice how long the check took: that is the cost of trusting that task to AI.

I would not let you leave part two without the failure catalogue, because every one of these failures has a signature, and a signature means a defence. There are four that account for most of the damage.

Fabrication. The fluent lie. The model states a torque specification, a legal clause, a statistic with total confidence, and it invented it. Defence: treat generation as drafting, never as record. Anything that goes to a customer, a regulator, or a court gets verified by a person whose name is on it. You already run this rule with junior staff; run it with the machine.

Drift. The model was trained on last year’s world and your business changes underneath it. Demand patterns shift after your new product line; the fraud classifier’s criminals adapt. A model that performed well in March quietly degrades by December. Defence: monitoring is part of the project, not an optional extra. Someone owns the question “is it still right?” and a trigger exists to retrain. When you evaluate a provider in lesson thirteen, this is question four.

Data debt. The pilot that worked on five hundred clean examples and collapsed on the real archive of ninety thousand messy ones. The spreadsheet in somebody’s laptop that nobody updated since the reorganisation. Defence: the assessment’s data pillar exists to catch this before you spend, and lesson eleven’s selection rule, pick problems whose data already exists, is the same defence from the other side.

The pilot-to-production gap. The most expensive failure, because it looks like success right up until the end. The demonstration works beautifully; the notebook glows; then nothing ships, because nobody planned who runs it, where it runs, what happens when it is wrong, and who maintains it after the consultants leave. Our own project records name this precisely: across three hundred and fifty delivered projects, the ones that moved a number ran inside whole workflows, in the business, used by the staff every day. What stayed isolated stayed a demonstration. Point solutions produce point gains. Defence: the project success definition from lesson six, agreed in writing before the build starts: the result runs inside your systems, used by your staff, or it does not count.

There is a fifth, quieter failure that is not technical at all: abandonment after the first stumble. An assistant hallucinates once, and the team quietly stops using it; nobody reassigns the task, nobody retrains the prompt, and eighteen months later the subscription is still being paid. Technology adoption in a small business is a leadership behaviour before it is a technical one, which is why the assessment weights leadership and culture the way it does, and why the boss using the tools openly matters more than any policy memo.

None of these failures is a reason to stay out. They are the reasons to enter the way this course has sequenced it: assess, then literacy, then one bounded project with a production bar. The catalogue is survivable with the defences in place. Without them, you will pay for one of these failures to teach you.

Try this. Take the last AI output you accepted without checking, an email draft, a summary, a suggestion. Check it now, properly. Whatever you find or do not find, notice how long the check took: that is the cost of trusting that task to AI.

Two islands, demo and production, separated by a broken bridge labelled who runs it, who maintains it — The most expensive failure looks like success until the end.