You can spot a pilot-purgatory agent program by its artifacts. A slick demo video. A Notion page called 'North Star Use Case.' A small team that has been 'three weeks from production' for six months. No customer is using it. No workflow has been retired. The agent is very impressive — in a sandbox.
The problem is rarely the model. The problem is almost always that the agent was scoped like a feature and built like a science project. Production-grade agents look and feel different from the start.
The five shifts that move agents into production
1. Replace 'do anything' with a single, boring job
The first instinct of a new agent team is to build the most capable thing possible. The capable version is impressive in a demo and useless in production because every edge case becomes a ticket. A narrow agent — 'summarize this ticket and suggest a disposition' — can be evaluated, governed, and owned. A wide agent can only be admired.
2. Treat the runtime as a product, not a script
A production agent runtime has retries, timeouts, idempotency, a cost ceiling, an escape hatch to a human, and structured traces for every step. If your agent is a single loop in a Python notebook, it is a prototype no matter how good the output looks.
3. Write evals before you write new prompts
When the team hits a regression, the instinct is to rewrite the prompt. That creates a short-term win and a long-term maze. Build an eval set — ideally drawn from real production traffic, with expected outputs or grading rubrics — and require every prompt change to run against it. Your best prompt engineer is a good eval.
4. Give the agent a named owner with on-call
Unowned agents drift. Someone needs to own the agent the way an SRE owns a service — with dashboards, alerting, a rotation, and authority to roll it back. If you cannot point at a single person responsible for this week's agent quality, the agent is not in production.
5. Ship with a measurable business metric
'AI-powered assistant launched' is not a metric. 'Average handle time down 32 seconds, CSAT unchanged' is. Pick a metric that the business already cares about and instrument the agent's deployment against it. If you cannot measure the impact, you cannot defend the investment at the next budget review.
A useful litmus test
Ask your agent team three questions. First — what will break if this agent is wrong 1% of the time? Second — who gets paged when it is down? Third — what number in the weekly business review moves because this agent exists? If you cannot get three crisp answers, the agent is not ready for production, no matter how long it has been in pilot.
Pilot purgatory is not a model problem or even a technology problem. It is a scoping and ownership problem that models have made easier to hide. The teams getting agents into production look less like research groups and more like platform teams — they ship smaller things more often and they have someone to call at 2am.
