Skip to content

Deployment

The pilot worked. 5 reasons it never shipped.

A demonstration that works is the cheapest part of the project. What stops it between the demo and production is almost never the model.

2 tan leather sofas facing a working table in a bright loft
A working room. Chapter sessions run 12 to 16 to a table with one agenda and no deck.
Published
Series
Deployment
Reasons examined
5
Questions before you start
6
About the model
None

The pattern

The demo is the cheapest part.

A pilot is built to answer one question: can the system produce a good output. It almost always can, and that answer costs weeks. Production asks a different set of questions, none of which the pilot was designed to answer, and answering them costs quarters.

What follows is 5 failures written as they appear from inside an operating company rather than from a vendor deck. None of them is a modelling problem. All 5 are decidable before a pilot begins, which is why the list at the foot of this note is the useful half of it.

The 5

None of them is the model.

A dark table behind a glass wall, an empty brick hall beyond it
A dark table behind a glass wall. Every one of these 5 is settled at a table rather than in the model.

  1. Nobody owns the output.

    How it presents
    The pilot has an owner and the output does not. Somebody sponsored the project. Nobody has agreed to be accountable for what the system decides once it is deciding.
    Where it stops
    At the first review that asks who signs off. The project does not fail, it stalls, and a stalled project is harder to kill than a failed one because nothing about it is wrong.
    What fixes it
    Name the accountable person before the build, and make it somebody whose existing job the output already belongs to. Every model risk framework in the country converges on this, so it is coming regardless.

  2. Legal arrived at the end.

    How it presents
    The system is built, the business case is signed, and review is booked as a formality 2 weeks before launch.
    Where it stops
    On the questions in the first note of this series: what is disclosed to the person, who can overturn the decision, what trained it and under what consent. Answering those retroactively usually means changing the architecture.
    What fixes it
    Review at the design stage, where each answer is a constraint rather than a rebuild. Legal review is a project phase with a duration and belongs in the plan with one.

  3. The unit economics were never run.

    How it presents
    The pilot ran on a budget nobody scaled. Cost per call, per document or per decision was irrelevant at pilot volume and is the entire business case at production volume.
    Where it stops
    At the finance review, at exactly the point the project looks most certain to proceed. This is the failure that arrives latest and costs most.
    What fixes it
    Price the production volume before the pilot rather than after it. Reserved capacity and committed spend are procurement decisions with their own timelines, and a CFO meeting them for the first time at launch will delay the launch.

  4. The first hire is the person who built it.

    How it presents
    The pilot works, so the person who built it is asked to run it. It is the obvious decision and it is usually the wrong one.
    Where it stops
    6 to 12 months later, quietly. Building a system that demonstrates well and operating one that fails safely at volume are different jobs, and the second is nobody's idea of a promotion.
    What fixes it
    Decide the operating role before the pilot proves anything, and staff it for monitoring, escalation and review rather than for model work. The builder stays on the build.

  5. There is no path for the wrong answer.

    How it presents
    The system is measured on how often it is right. Nothing is designed around the times it is not, because at pilot volume somebody noticed and fixed it by hand.
    Where it stops
    On the first output that reaches a customer, a regulator or a court. At that point the absence of a documented review path is the finding, not the error itself.
    What fixes it
    Design the escalation before the accuracy target. A person who can be reached, who holds the information the decision was made on, and who has the authority to reverse it.

Before you start

6 questions that would have caught all 5.

A long white table under tall windows, 16 chairs and nobody in the room
A long table under tall windows, 16 chairs. A chapter session seats 12 to 16 and works one agenda.

  1. Whose existing job does this output belong to?

    If the honest answer is nobody, the project has a sponsor and no owner, and 01 is already in motion.

  2. What does this cost per decision at 100 times pilot volume?

    Asked before the build rather than at the finance review, where it becomes a reason to stop.

  3. What do we have to tell the person the decision is about?

    A design constraint if it is asked now and an architecture change if it is asked in review.

  4. Who reverses a wrong answer, and how do they hear about it?

    Both halves. An authority with no notification path is the same as no authority.

  5. Who operates this in 12 months, and is it the builder?

    If it is the builder, ask whether they want the job. The answer is usually no and is rarely sought.

  6. Which of our customers inherit an obligation from this?

    Banks, hospitals and ministries carry requirements down the contract. That surface is the first note in this series.

Read next

The rest of the method.

  1. Governance

    There is no federal AI law

  2. Capital

    Where Canadian AI capital is

The rest of this is said in a room.

A note publishes the method. What it cannot publish is the 2 hours of argument that produced it, which is the part membership is actually for.

Request an Invitation