Here’s a pattern I keep seeing: an AI project looks most impressive right before anyone really understands it.
The demo has one job — make people believe. One input, one output, one “wow” moment. Feed it a messy PDF, watch it spit out clean data. Ask it a question, get a smart answer. Everyone in the room leans in.
Then someone tries to ship it. And the cracks show up fast.
What actually happens after the demo
Say you built a bot that reads invoices and pulls out the total, vendor, and due date. In the demo, you tested it on ten clean PDFs. It nailed all ten.
Three weeks later, in production:
- A scanned invoice comes in sideways. The model reads the total as $0.
- Two vendors use the same layout but different currency symbols. It picks USD both times.
- Someone emails a screenshot instead of a PDF. The pipeline chokes.
- The finance team starts double-checking every output — which defeats the point of automating it.
Nothing here means the model is bad. It means the demo never tested for reality. A demo answers “can this work once?” Production asks “how often does this fail, and what happens when it does?” Those are different questions, and only one of them tells you if you have a product.
Stop treating the model as the whole product
It’s easy to think the model is the product. It isn’t. It’s one part sitting inside a bigger workflow — and the workflow is what the user actually experiences.
Take a support-ticket triage system. Say the model correctly labels tickets 90% of the time. Sounds great. But the product still falls apart if:
- Nobody set a confidence threshold, so uncertain guesses get treated the same as sure ones.
- There’s no way to route a low-confidence ticket to a human.
- Agents can’t correct a wrong label in two clicks — so they just work around the tool.
- Nobody’s watching whether ticket topics have shifted since last month.
None of that is a “the model needs to be smarter” problem. It’s a workflow that was never designed to handle the model being wrong sometimes — which it will be.
So before picking a model, ask a more boring question: what decision is this actually supposed to improve? If you can’t answer that clearly, you’ll end up tweaking prompts and comparing benchmarks forever, with no way to know if any of it matters.
Write down how it will fail — before it does
Normal software fails loudly. A button does nothing. A page 404s. You know something’s wrong immediately.
AI fails quietly. It gives you a confident, fluent, completely wrong answer — and it can look identical to a correct one. That’s the dangerous part.
So before launch, sit down and actually list the ways this can go wrong. For the invoice bot, that table might look like:
| What breaks | What it looks like | Why it matters | What you do about it |
|---|---|---|---|
| Wrong number pulled | $450 read as $4,500 | Someone gets overpaid | Flag anything above a threshold for human check |
| Bad scan quality | Blurry or rotated PDF | Missing data, silent failure | Reject it with a clear message instead of guessing |
| Made-up detail | Invents a PO number that doesn’t exist | Breaks trust in the whole tool | Cross-check against known records before accepting |
| Unclear request | Invoice in a format you’ve never seen | Garbage output | Route to a human, don’t force an answer |
| Cost creep | Someone uploads a 200-page contract | Surprise API bill | Cap file size, batch large jobs |
This isn’t paperwork for its own sake — it’s the actual design work. Skip it, and you’ll discover these failure modes one angry customer at a time, which is the most expensive way to learn anything.
Test it before you trust it
A lot of teams build the thing, eyeball a handful of outputs, decide it “seems good,” and ship. Then six weeks later nobody can say whether the last change made things better or worse.
Build a test set before you polish anything. It doesn’t need to be huge — even 30-50 well-chosen examples beats zero. What matters is that it’s honest, not flattering. For the invoice bot, that means including:
- A few clean, easy invoices (the happy path — but not the only path)
- A blurry scan
- An invoice missing a due date entirely
- One in a foreign currency
- One where the answer genuinely isn’t in the document
Run every model or prompt change against this same set. Now you can actually see if you’re improving or just moving the errors around.
Human review isn’t a failure — it’s often the whole point
There’s a real temptation to chase “fully automated” because it sounds more impressive. But full automation is rarely the right first goal. Often the better outcome is: the AI does the boring 80%, and a person handles the judgment calls.
The question isn’t “can we remove the human?” It’s “where does a human’s judgment actually matter most?”
Sometimes that means: the model handles easy cases and hands off anything uncertain. Sometimes it means the model drafts a reply and a person hits approve. Sometimes the model just gathers the evidence and a human still makes the call. None of these are compromises — they’re often what makes people actually trust and use the tool.
Things will change under you — plan for that
Even a system that works well on day one will drift. Users find new ways to use it. Document formats change. The vendor updates the underlying model without asking you. What counted as a normal request in January might be rare by June.
So don’t just watch if the system is running. Watch if it’s still useful:
- How much is it costing per request, and is that changing?
- How often are people overriding or rejecting its output?
- Are the inputs it’s seeing starting to look different from what you tested?
- Is it quietly getting slower?
Uptime monitoring tells you the lights are on. None of that tells you whether the thing is still doing its job.
The better question to ask before you build anything
Don’t ask “can we use AI for this?” That question makes every idea sound viable.
Ask instead: what decision or bottleneck are we actually trying to fix, and how wrong can this system be before it stops being worth it?
That single reframe forces the real questions to show up early — what counts as success, what failure looks like, who reviews the edge cases, what you’re tracking after launch. Here’s a quick gut-check before you take any demo seriously:
- Can you say, in one sentence, what decision this improves?
- Do you know what a bad output looks like, specifically?
- Do you have a test set with real, messy examples — not just the easy ones?
- Is there a fallback when the model doesn’t know?
- Have you decided where a human needs to be in the loop?
- Do you know what this costs and how fast it responds, under real load?
- Is someone actually going to watch this after it ships?
If most of those are blank, that’s fine — it just means you have a good demo, not a product yet. The gap between the two isn’t more AI. It’s the unglamorous work of deciding what “reliable” actually means, and building the workflow around the model instead of just trusting the model itself.