An AI Agent Development Roadmap That Reaches Production
The failure pattern for agent projects is not technical. It is a good prototype in week three, followed by three months of undirected improvement, followed by a quiet cancellation because nobody could say whether it was ready.
This roadmap fixes that with exit tests. Each phase has a condition that is either met or not, written down before the phase starts. If a phase cannot pass its test, you either fix the specific gap or stop — and both of those are better outcomes than drifting.
Phase 1 — Scope (1–2 weeks)#
Write the task down as a sentence a reviewer could mark right or wrong. List the tools with their argument schemas. Decide which actions are irreversible and will sit behind a human. Collect twenty real examples from logs or from the person who does this job today, including the awkward ones.
Exit test: a colleague who was not in the meetings can read the brief and correctly grade five example runs.
Phase 2 — Prototype (2–3 weeks)#
Build the smallest loop that does the task with real tools against a test environment. No framework decision yet if you can avoid it; a hand-written loop teaches you what you actually need. Run your twenty cases, look at every failure, and fix the tool design rather than the prompt where you have the choice.
Exit test: 60% of the twenty cases pass end to end, and every failure has an identified cause written next to it.
Phase 3 — Harden (3–5 weeks)#
This is where most of the real work lives, and where under-budgeted projects die. Permissions on the end user identity, approval gates on irreversible actions, argument validation, tracing you can read, monitoring with a handful of behavioural metrics, the evaluation set grown to fifty cases including adversarial ones, and a degraded mode for when the provider misbehaves.
- Authorisation checked server-side on every tool call.
- Human gate in front of every irreversible action, with decision-ready context.
- Step and spend caps, plus repeat detection.
- Traces with a run identifier on every call, and a retention decision made deliberately.
- Adversarial cases in the evaluation set, run like any other test.
Exit test: 85% on the evaluation set, 100% on the should-refuse subset, and no unresolved security finding.
Phase 4 — Launch (2 weeks, then continuous)#
Start with a limited audience — one team, one customer segment, or a percentage of traffic. Read runs daily. Keep the escalation path obvious and staffed, because the first week produces surprises regardless of how good the evaluation set was. Widen when the numbers hold for two consecutive weeks, not when the calendar says so.
| Week | Audience | What you are watching |
|---|---|---|
| 1 | Internal team only | Traces, obvious failures, tool errors |
| 2 | 5% of real traffic | Escalation rate, success rate |
| 3–4 | 25% | Cost per task, latency at peak |
| 5+ | Full, if the numbers hold | Drift, new request categories |
What happens after launch#
An agent is a service, not a project. Someone owns it, the evaluation set keeps growing from real traffic, model versions get re-qualified when providers deprecate, and the human gates get selectively removed as evidence accumulates. Teams that plan the first four phases and nothing beyond them end up with an agent that was excellent in October and quietly wrong by March.
Frequently asked questions
Twelve weeks feels long for a demo that worked in three days.
The demo did work. The remaining nine weeks are permissions, evaluation, monitoring and the failure paths — the things that decide whether it can be trusted with real customers. Skipping them does not remove the work; it moves it to after the incident.
Can phases overlap?
Hardening can start during the prototype, and usually should for permissions. Do not start launch before the hardening exit test passes: a limited launch with no approval gates is not a limited risk, it is a small sample of an unlimited one.
What if the prototype fails its exit test?
Look at the causes you wrote down. If they are tool design and data access, fix them — that is normal. If they are that the task requires judgement nobody can define, stop. Stopping in week five is a good outcome; stopping in month seven is an expensive one.
ai agent roadmapagent project planllm project phasesai delivery timelineagent launch checklist
