Verify, Never Assert: Making AI Safe in Production Systems

Most conversations about AI in manufacturing get stuck on the wrong question. People ask whether the model is good enough. In our experience the model is the least interesting variable. What decides whether an AI agent is safe near a production system is the rules you put around it.

On a recent Windchill upgrade for an aerospace manufacturer, an AI agent executed roughly 90% of the steps. Forty of forty phases completed on the production weekend. No rollback was needed. That outcome came from two rules, and neither of them is about artificial intelligence.

Rule one: verify, never assert

The agent was never permitted to conclude that a step had worked. It had to prove it.

This sounds pedantic until you watch what goes wrong without it. An installer prints a success message. A script exits zero. A log line says "complete." None of those are evidence that the thing you wanted actually happened — they are evidence that a process believed it happened. The two diverge more often than anyone is comfortable admitting.

So the standing instruction was: never infer completion. Go and look. Read the log, query the database, list the directory, and bring back what you found. If the agent said a step was done, it had to attach the evidence that made it true.

The interesting side effect is that this rule is not really about AI. Experienced administrators get fooled by success messages too. Writing the rule down forced a discipline that humans are supposed to have and frequently skip at hour nine of a maintenance window.

Rule two: draw the line at reversibility, not difficulty

The instinct when deciding what to automate is to give the machine the easy work and keep the hard work for yourself. That instinct is wrong, and it is wrong in a way that costs you.

AI is good at hard things. It is good at reading a thousand lines of stack trace, at cross-referencing configuration across a dozen files, at grinding through a problem by trial and error far longer than a person will. Difficulty is not where it fails.

Where it fails is when being wrong is expensive.

An agentic workflow can try, fail, and try again almost indefinitely on a reversible step. That is a feature. But a database schema transformation cannot be untried. A vault move that desynchronises metadata from physical files cannot be casually undone across millions of objects. The cost of being wrong, not the difficulty of being right, is what should decide where the human sits.

So the line was drawn there. Every irreversible step was gated: the agent stopped, presented its state, and waited for a human instruction to proceed. Anything reversible, it owned.

What that looks like in practice

On this engagement the split ended up roughly like this. The agent planned and executed the operating system upgrade end to end, with a human verifying the outcome. It ran the scriptable, time-consuming, repetitive work — the parts that are tedious rather than risky. It monitored logs, stack traces, database state and directory contents in parallel, which is something a two-person team simply does not have the hours to do properly.

The human kept the graphical steps of the upgrade manager, and every irreversible call.

That division is not a statement about what AI is capable of. It is a statement about where a mistake is affordable.

The failure mode nobody plans for

The risk most teams prepare for is the agent not knowing something. That turns out to be the manageable one — a gap in knowledge announces itself.

The real risk is confident wrongness.

During one rehearsal, our architect was working through the upgrade manager interface, multitasking, taking his time. The agent was watching the console log. It saw nothing progressing, concluded the process had hung, and killed and restarted it.

Nothing malfunctioned. It reasoned its way to a plausible conclusion from incomplete information and acted on it before anyone looked up. That is exactly the behaviour that "verify, never assert" exists to catch, and exactly why irreversible steps sit behind a gate.

The uncomfortable prerequisite

There is a version of this article that ends with "and so any team can do this." That version would be dishonest.

The person supervising has to know the process well enough to catch a confident wrong answer. That is the whole prerequisite, and it is not negotiable. AI amplifies whoever is sitting in front of it. Put an experienced architect behind an agent and you get a genuine multiplier. Put someone without the domain knowledge behind the same agent and it will lead them somewhere no experienced person would go, with complete conviction the entire way.

Which means the sequence for a team wanting to work this way runs backwards from what most people expect. First find the experience. Then translate that experience into rules, runbooks and error catalogues. Then let the agent work inside them.

The knowledge base is the product. The model is a commodity.