# aiagentdevelopment.info — full text

> The complete text of every guide in this language, so an answer engine can read the catalogue in one request. Nothing here is absent from the visible pages.

## An AI Agent Development Roadmap That Reaches Production
https://aiagentdevelopment.info/guides/ai-agent-development-roadmap
Updated 2026-08-05 · Cost and business

- Four phases with written exit tests: scope, prototype, harden, launch.
- Hardening is the longest phase and the one most often under-budgeted.
- Launch to a limited audience and widen on numbers, not on the calendar.
- After launch it is a service: an owner, a growing eval set, re-qualification on model changes.

The failure pattern for agent projects is not technical. It is a good prototype in week three, followed by three months of undirected improvement, followed by a quiet cancellation because nobody could say whether it was ready.

This roadmap fixes that with exit tests. Each phase has a condition that is either met or not, written down before the phase starts. If a phase cannot pass its test, you either fix the specific gap or stop — and both of those are better outcomes than drifting.

### Phase 1 — Scope (1–2 weeks)
Write the task down as a sentence a reviewer could mark right or wrong. List the tools with their argument schemas. Decide which actions are irreversible and will sit behind a human. Collect twenty real examples from logs or from the person who does this job today, including the awkward ones.
Exit test: a colleague who was not in the meetings can read the brief and correctly grade five example runs.

### Phase 2 — Prototype (2–3 weeks)
Build the smallest loop that does the task with real tools against a test environment. No framework decision yet if you can avoid it; a hand-written loop teaches you what you actually need. Run your twenty cases, look at every failure, and fix the tool design rather than the prompt where you have the choice.
Exit test: 60% of the twenty cases pass end to end, and every failure has an identified cause written next to it.

### Phase 3 — Harden (3–5 weeks)
This is where most of the real work lives, and where under-budgeted projects die. Permissions on the end user identity, approval gates on irreversible actions, argument validation, tracing you can read, monitoring with a handful of behavioural metrics, the evaluation set grown to fifty cases including adversarial ones, and a degraded mode for when the provider misbehaves.
- Authorisation checked server-side on every tool call.
- Human gate in front of every irreversible action, with decision-ready context.
- Step and spend caps, plus repeat detection.
- Traces with a run identifier on every call, and a retention decision made deliberately.
- Adversarial cases in the evaluation set, run like any other test.
Exit test: 85% on the evaluation set, 100% on the should-refuse subset, and no unresolved security finding.

### Phase 4 — Launch (2 weeks, then continuous)
Start with a limited audience — one team, one customer segment, or a percentage of traffic. Read runs daily. Keep the escalation path obvious and staffed, because the first week produces surprises regardless of how good the evaluation set was. Widen when the numbers hold for two consecutive weeks, not when the calendar says so.
| 1 | Internal team only | Traces, obvious failures, tool errors |
| 2 | 5% of real traffic | Escalation rate, success rate |
| 3–4 | 25% | Cost per task, latency at peak |
| 5+ | Full, if the numbers hold | Drift, new request categories |

### What happens after launch
An agent is a service, not a project. Someone owns it, the evaluation set keeps growing from real traffic, model versions get re-qualified when providers deprecate, and the human gates get selectively removed as evidence accumulates. Teams that plan the first four phases and nothing beyond them end up with an agent that was excellent in October and quietly wrong by March.

Q: Twelve weeks feels long for a demo that worked in three days.
A: The demo did work. The remaining nine weeks are permissions, evaluation, monitoring and the failure paths — the things that decide whether it can be trusted with real customers. Skipping them does not remove the work; it moves it to after the incident.

Q: Can phases overlap?
A: Hardening can start during the prototype, and usually should for permissions. Do not start launch before the hardening exit test passes: a limited launch with no approval gates is not a limited risk, it is a small sample of an unlimited one.

Q: What if the prototype fails its exit test?
A: Look at the causes you wrote down. If they are tool design and data access, fix them — that is normal. If they are that the task requires judgement nobody can define, stop. Stopping in week five is a good outcome; stopping in month seven is an expensive one.

## AI Agent Use Cases by Industry: What Is Actually Working
https://aiagentdevelopment.info/guides/ai-agent-use-cases-by-industry
Updated 2026-08-05 · Cost and business

- Agents stick where the task is narrow, a system of record exists, and risky steps are gated.
- Internal reconciliation, triage and drafting are the most reliable first wins.
- Projects stall from missing APIs, no definition of done, or no owner — not from model limits.
- In regulated sectors, start on the drafting side and widen autonomy on evidence.

Use-case lists usually read like a wish list. This one is drawn from what we see reaching production and staying there — which is a much shorter list, and a more repetitive one than the marketing suggests.

The pattern is consistent across industries. A narrow task with a clear definition of done, two or three tools against a real system of record, and a person in front of whatever cannot be undone. Where those three hold, agents stick. Where they do not, projects stall in pilot regardless of sector.

### What is working, by sector
| E-commerce | Order status, returns eligibility, address changes | Order lookup, returns policy, address update | Refunds above a threshold |
| SaaS support | Tier-one triage with account context | Account lookup, docs search, ticket update | Plan changes, credits |
| Finance ops | Invoice matching against purchase orders | ERP read, document parse, flag exception | Any payment |
| Healthcare admin | Appointment scheduling and reminders | Calendar, patient record read | Anything clinical |
| Recruiting | Screening against explicit criteria, scheduling | ATS read, calendar, email draft | Rejections and offers |
| Logistics | Exception handling on delayed shipments | Tracking, carrier API, customer notify | Compensation offers |

### The internal agents nobody writes about
The most reliable wins are unglamorous and internal: reconciling two systems that disagree, drafting the first version of a recurring report, triaging inbound requests into the right queue with the right context attached, and answering employee questions about policy with a citation. They work because the definition of done is clear, the audience tolerates an imperfect first draft, and mistakes are cheap and visible. They are also where teams learn the operational habits that a customer-facing agent will require.

### Where projects stall, in every sector
- No system of record with an API — the agent has nothing solid to stand on.
- No agreed definition of a correct outcome, so nobody can grade it.
- The task is judgement-heavy but the risk appetite is zero, so every action is gated and the value disappears.
- Ownership is unclear: built by innovation, needed by operations, on call to nobody.

### Choosing your first one
Rank candidate tasks on four axes: volume, how repetitive the steps are, whether a system of record exists, and how reversible the actions are. The best first project is high volume, highly repetitive, backed by an API, and reversible. That is rarely the most impressive-sounding idea in the room, and it is almost always the one that reaches production and funds the next one.
Deliberately pick something where a mistake is embarrassing rather than expensive. Your first agent is also how your organisation learns to trust the category.

### Regulated sectors: slower, not closed
Finance, healthcare and the public sector can absolutely run agents; they simply have to start on the drafting side of the line. An agent that assembles a case, cites its sources and hands a person a decision-ready summary delivers most of the time saving with none of the automated-decision exposure. Once the audit trail is proven and the numbers are boring, the conversation about widening autonomy becomes a normal one — and it starts from evidence rather than from a promise.

Q: Which industry has the clearest wins?
A: E-commerce and SaaS support, because the tasks are high volume, the systems of record have decent APIs, and most actions are reversible. That combination is what makes an agent easy to justify, not anything about the sector itself.

Q: Are agents useful for small businesses?
A: Yes, usually in the internal-operations shape: triage, drafting, reconciliation. The constraint is the same as for large ones — if the data lives only in spreadsheets and inboxes, fix access first or the agent will be guessing.

Q: How do I estimate value before building?
A: Count the task volume, measure how long a person takes today, and estimate the share the agent can complete without help. Be conservative on that share; a first agent that completes 60% of a high-volume task is a strong result and a much safer promise than 95%.

## Measuring the ROI of an AI Agent Without Fooling Yourself
https://aiagentdevelopment.info/guides/measuring-roi-of-ai-agents
Updated 2026-08-05 · Cost and business

- Write the baseline down before launch — volume, handling time, cost, current error rate.
- Count completed tasks that needed no human, not deflection or messages handled.
- Subtract error cost and review time; that subtraction is what makes the number credible.
- Report qualitative benefits separately rather than converting them into invented money.

Most agent ROI figures do not survive a careful reading, and the reason is almost always the same: the baseline was reconstructed after the fact, and the errors were left out of the arithmetic.

Getting an honest number is not hard, but it has to start before launch. Write down what today costs, in the units you will use afterwards. Everything else follows from that one discipline.

### Write the baseline down first
- Volume: how many of these tasks happen per week?
- Handling time: how long does a person take, measured on a sample rather than remembered?
- Fully loaded cost per hour for the people doing it.
- Current quality: error or rework rate today, because the agent will be compared against it.
- Waiting time: how long the requester currently waits, if that matters to them.
A baseline reconstructed after launch will always favour the project, and everyone reviewing it knows that.

### Metrics that hold up, and the ones that flatter
| Deflection rate | Counts unanswered as resolved | Completed tasks needing no human |
| Messages handled | Volume is not value | Tasks completed end to end |
| Satisfaction on agent chats | Survivor bias; the frustrated leave | Satisfaction across all contacts |
| Time saved per response | Ignores review time | Net minutes after human review |
| Cost per token | Not a business number | Cost per completed task |

### The formula, including the part people omit
Annual benefit equals tasks completed without a human, times minutes saved per task, times loaded cost per minute — minus the cost of errors the agent introduced, minus the review time it created. That subtraction is the honest part. An agent that completes 70% of tasks but requires a human to check every one has saved review time, not handling time, and the difference is usually a factor of three.
Estimate error cost explicitly, even roughly: correction time, plus goodwill, plus any refund or credit issued because of a mistake.

### Benefits that are real but not on the invoice
Some genuine value never appears in the cost model. Faster responses at three in the morning. Consistency — the same answer to the same question regardless of who is on shift. A written trace of why something was decided, which is worth a great deal in a regulated environment. Staff spending their time on the interesting half of the work. Report these separately and honestly rather than converting them into invented currency; a finance reviewer trusts a stated qualitative benefit far more than a suspiciously precise number.

### When the honest answer is no
Sometimes the arithmetic says stop, and saying so is the most valuable thing an ROI exercise does. Low volume tasks rarely pay back a build. Tasks where every output must be checked anyway save review time only. And tasks where errors are expensive can produce a negative return even at high accuracy — 3% wrong on ten thousand high-value decisions is three hundred problems. Publish that result too. A team that has killed one agent on evidence is much more credible the next time it proposes one.

Q: What is a realistic completion rate for a first agent?
A: Sixty to eighty percent of a well-scoped, high-volume task, with the remainder escalated. Anyone promising ninety-five percent before seeing your data is describing a benchmark, not your Tuesday afternoon.

Q: How long until payback?
A: For a well-chosen internal task at reasonable volume, commonly six to twelve months including maintenance. If your model shows payback in six weeks, check whether error cost and review time are in the arithmetic.

Q: How do I count value when the agent only drafts?
A: Measure the time from blank page to approved output, before and after. Drafting agents frequently deliver most of the saving with a fraction of the risk, and they are much easier to get approved in the first place.

## Hiring AI Agent Developers: What to Look For and How to Test
https://aiagentdevelopment.info/guides/how-to-hire-ai-agent-developers
Updated 2026-08-05 · Cost and business

- Hire systems engineers who think in failure modes, not prompt specialists.
- Screen with a brief: tool schemas, stopping conditions, ten eval cases, human gates.
- Framework knowledge is the least predictive signal of success.
- Require handover of prompts, schemas, evaluation set and traces from any supplier.

The job title is new, the skill set is not. The people who build agents that survive production are ordinary strong engineers who have learned to work with a component that is fast, capable and occasionally confidently wrong.

That reframing makes hiring much easier. You are not looking for a prompt specialist. You are looking for someone who instinctively asks what happens when the tool returns nothing, and who has an opinion about how you would know the change made things better.

### What matters, in order
- API and integration engineering: most of the work is talking to your systems properly.
- Testing instinct: they ask about evaluation before they ask about the model.
- Failure-mode thinking: empty results, permissions, timeouts, partial success.
- Security awareness: least privilege, injection, audit trails, approval gates.
- Cost awareness: they can explain where tokens go without looking it up.
- Model familiarity: useful, learnable in weeks by a strong engineer.
- Framework knowledge: the least important item, and the most advertised.

### A ninety-minute screening exercise that works
Give a candidate a short brief: an agent that answers order questions and can issue refunds under fifty pounds. Ask for the tool list with argument schemas, the stopping conditions, ten evaluation cases, and what sits behind a human gate. You are not looking for code. You are looking for whether they define `refund_order(order_id)` rather than `update_order(order_id, fields)`, whether they include a should-refuse case and an empty-result case, and whether the refund gate appears without being prompted.
Strong candidates ask clarifying questions about permissions and edge cases in the first five minutes. It is the most reliable signal in the whole process.

### Questions that separate experience from enthusiasm
| How do you know a change helped? | We test it manually | A fixed evaluation set, run before and after |
| What do you do when a tool returns nothing? | Retry | Return an explicit empty result the agent can act on |
| How do you stop injection? | Instruct the model to ignore it | Least privilege, content isolation, approval gates |
| Why was your last agent slow? | The model was slow | Six serial calls; parallelised two, cut context |
| How do you pick a model? | The best one | Per step, measured on our own cases |

### Agency, contractor or in-house
An agency suits a first build with a deadline: you buy a team that has already made the standard mistakes, and you should require handover of the evaluation set and tool schemas as a deliverable. A contractor suits extending a system your team will keep. In-house is right when the agent becomes part of the product — at that point someone must own it permanently, and that person needs the context that only comes from building it. The common failure is an agency build with no handover, leaving a system nobody internally can change.

### Red flags on both sides
- A proposal with no evaluation line, or where evaluation means the developers trying it.
- Confidence about accuracy before seeing your data.
- A framework recommendation before the tool list is written.
- No question about permissions or who the end user is.
- Unwillingness to hand over prompts, schemas and evaluation cases at the end.

Q: Do I need a machine learning engineer?
A: Usually not. Agent work is systems engineering against a model API. Bring in ML expertise when you are fine-tuning, training a classifier, or doing serious retrieval optimisation — not to build the agent itself.

Q: How big should the team be?
A: Two engineers and a part-time domain expert covers most first builds. The domain expert is not optional: they supply the evaluation cases and decide what a correct outcome looks like, and no amount of engineering substitutes for that.

Q: What should an agency hand over?
A: Repository, prompts, tool schemas, the evaluation set with results, traces from the last month, a monitoring dashboard, and a written note of known failure modes. If any of these are missing, you have bought a system you cannot safely change.

## AI Agent Development Cost: Real Numbers and What Drives Them
https://aiagentdevelopment.info/guides/ai-agent-development-cost
Updated 2026-08-05 · Cost and business

- Internal agents commonly cost $8k–$45k to build; customer-facing $35k–$150k.
- Integrations, evaluation and permissions dominate the hours — prompts are the smallest line.
- Run cost is usually modest and halves with routine context and routing work.
- Budget 15–25% of build cost per year for maintenance, and name an owner.

Nobody can quote your project from a web page, but the ranges are not a mystery either, and the shape of the estimate is remarkably consistent across the builds we have done and reviewed.

The three numbers you need are build, run and maintain. Teams routinely negotiate hard on the first, worry about the second, and forget the third entirely — which is why so many agents are quietly broken eight months after launch.

### Build cost, by scope
| Internal assistant, 2–3 read-only tools | $8k–$20k | Loop, tools, retrieval, small eval set |
| Internal agent with write access | $20k–$45k | Above plus permissions, audit, approval gates |
| Customer-facing support agent | $35k–$90k | Above plus escalation, tone work, monitoring, load |
| Agent inside your product | $60k–$150k+ | Above plus UI, multi-tenancy, SLA, versioning |
| Proof of concept only | $5k–$12k | One path, no permissions, not shippable |

### Where the hours actually go
The distribution surprises people who expect the model to be the project. Across builds, roughly: integrations and the tool layer 30%, evaluation and iteration 20%, permissions, audit and security 15%, monitoring and operational tooling 10%, prompt and retrieval work 15%, and the agent loop itself around 10%. Prompt engineering is the smallest line on the sheet, which is exactly why a quote that consists mostly of prompt engineering is a warning sign.
If a proposal has no line for evaluation, you are buying a demo. The evaluation set is what turns a demo into something you can change without fear.

### Run cost is usually smaller than feared
For a typical support-style agent, a completed task costs somewhere between a few cents and a few tens of cents in model calls, depending on context size and how many steps it takes. At ten thousand tasks a month that is real money but rarely the dominant number next to the labour it displaced. It also falls quickly with the standard measures — trimming context, routing simple steps down a tier, caching a stable prefix — which together commonly halve the bill without touching output quality.

### Maintenance is the line people forget
- Model deprecations: re-qualify on a new version once or twice a year.
- API drift: the systems your tools call will change without asking you.
- Retrieval upkeep: documents change, and a stale index is worse than none.
- Evaluation growth: new failure categories arrive with new users.
- Ownership: someone must be on call when the agent does something odd.
Budget 15–25% of build cost per year. An agent is a production service, not a project that finishes.

### How to get a quote you can compare
Ask every supplier for the same five things and the numbers become comparable: the tool list with argument schemas; who builds the evaluation set and how many cases; which actions sit behind human approval; what monitoring is delivered; and what a maintenance retainer covers. Ranges that differ by a factor of three almost always turn out to be quoting different scopes — one includes permissions and evaluation, the other is a demo with a nice interface.

Q: Why do quotes differ so much for the same brief?
A: Because the brief is rarely as specific as it feels. A quote covering permissions, evaluation, monitoring and a maintenance path is a different product from one covering a working happy path. Compare the five items above rather than the headline number.

Q: Can we start smaller than these ranges?
A: Yes. Pick one narrow task, two read-only tools and twenty evaluation cases; that is often $8k–$15k and tells you whether the larger build is worth funding. It also produces the tool layer and evaluation harness the bigger project would need anyway.

Q: Is it cheaper to build in-house?
A: Cheaper in cash, more expensive in time, and only if someone senior owns it. The usual failure is a promising internal prototype that nobody has time to take through permissions, evaluation and monitoring — which is where most of the cost was hiding.

## Scaling AI Agents: Latency, Concurrency and Rate Limits
https://aiagentdevelopment.info/guides/scaling-ai-agents
Updated 2026-08-04 · Production and ops

- The bottleneck is provider quota and multi-second calls, not your servers.
- Split interactive from background traffic and queue the background class.
- Stream, show real progress, and return useful partial results at caps.
- Build a degraded mode behind a switch before an incident forces you to.

The first traffic spike teaches every team the same lesson. Your servers are nearly idle, your database is fine, and everything is slow — because each request is several multi-second calls to a provider that has a quota, and quotas do not care how many containers you started.

Scaling agents is therefore mostly queueing theory and expectation management, with a little capacity planning. The good news is that the techniques are well understood and none of them require rewriting your agent.

### Know which of the three limits you are hitting
| 429 responses from the provider | Requests or tokens per minute | Queue, backoff, spread across keys or regions |
| Slow but no errors | Serial model calls per run | Parallelise independent steps; shorten the loop |
| Memory or connection exhaustion | Your own service | Ordinary capacity work |
| Slow only at peak hours | Shared quota contention | Priority queue; shed low-value work |

### Queue everything that is not interactive
Split traffic into two classes on day one. Interactive work — a person waiting — gets a short deadline, a strict step cap, and a fast model where quality allows. Background work — batch classification, enrichment, overnight processing — goes into a queue with concurrency you control, and it is the first thing you throttle when quota gets tight. Without this split, a batch job started at nine in the morning becomes an outage for people using the product.

### Make the wait feel shorter, honestly
- Stream the answer as it is produced rather than after the last token.
- Show the current step in plain language: `checking your order`, not a spinner.
- Return the useful partial result when a cap is hit, with what is missing named.
- Move anything non-blocking off the critical path and deliver it afterwards.
Latency perception is a product problem as much as an engineering one. A five-second answer with visible progress beats a three-second blank screen in every test we have run.

### Design the degraded mode before you need it
Decide in advance what the agent does when the provider is slow, over quota, or down — and build it while you are calm. A sensible ladder: full agent, then a cheaper or alternative model, then retrieval-only answering with no tool calls, then a plain apology with a handover to a human. Put it behind a switch an on-call engineer can flip in seconds. Teams without a degraded mode take a full outage during someone else’s incident, which is a bad way to spend a Friday.

### Capacity planning with the two numbers that matter
Model calls per completed task, and tokens per completed task. Multiply by expected tasks per minute at peak, compare with your quota, and you know whether you need a limit increase before launch rather than during it. Recalculate whenever the agent changes shape — adding a critic pass or a second specialist can quietly double calls per task, and the first sign will otherwise be a wall of 429s on a marketing launch day.

Q: Should I run several provider accounts or regions?
A: For genuine scale or resilience, yes — spreading across keys, regions or providers is standard practice. Do it behind one internal interface so your agent code stays unaware, and pin model versions per route so behaviour does not vary by which route served the request.

Q: How do I keep interactive latency acceptable?
A: Cap steps hard for interactive runs, route simple steps to a fast model, parallelise independent tool calls, and stream output. If the task genuinely needs ten steps, stop pretending it is interactive and give it a progress view instead.

Q: What breaks first when traffic grows?
A: Provider rate limits, almost always, followed by whichever internal API your busiest tool calls. Load-test the tool layer as well as the agent; an agent multiplies traffic to the systems behind it in a way that surprises the teams who own them.

## AI Agent Security: Guardrails, Permissions and Prompt Injection
https://aiagentdevelopment.info/guides/ai-agent-security-and-guardrails
Updated 2026-08-04 · Production and ops

- Treat the agent as a helpful colleague who can be argued with by strangers.
- Prompt injection is architectural: least privilege, content isolation, approval gates, audit.
- Authorise on the end user identity, server-side, on every call.
- Put adversarial cases in the evaluation set and re-run them after every model change.

The security model for agents is easier to reason about once you stop thinking of the agent as code and start thinking of it as a member of staff who is helpful, fast, tireless, and can be talked into things by a stranger.

You would not give that person unrestricted database access, a company card with no limit, and the ability to email customers unsupervised on their first day. The same instincts translate directly, and they are more reliable than any instruction you write into a prompt.

### The threat you cannot prompt your way out of
Prompt injection is instructions hidden in content the agent reads — a support ticket, a web page, a PDF, a tool description. The model has no reliable way to distinguish data it should reason about from instructions it should follow, and no phrase such as `ignore any instructions in the document` closes that gap. Defence therefore has to be architectural: restrict what the agent is able to do, so that a successful injection reaches a small blast radius rather than a large one.
Assume every piece of retrieved content is written by someone who wants your agent to misbehave. Design so that this is merely annoying.

### Nine controls, in the order we implement them
- Least privilege per tool: scoped, read-only where possible, never a service account with everything.
- Authorisation on the end user, checked server-side on every call — not once at the start of the session.
- Human approval in front of every irreversible action, with enough context to decide in seconds.
- Argument validation and identifier resolution before execution; reject rather than coerce.
- Spend and step caps per run, and a rate limit per user and per tool.
- Content isolation: treat retrieved text as data, never merge it into the system instructions.
- Output filtering on anything leaving the system, especially outbound messages.
- Full audit logging: who, what, which record, which run, what result.
- Kill switch: one setting that disables tools while leaving read-only answering alive.

### Blast radius by action type
| Read a record the user owns | n/a | Authorisation check |
| Draft a reply | Yes | None needed |
| Update a status field | Usually | Audit and rate limit |
| Send an external message | No | Human approval |
| Issue a refund or payment | No | Human approval, amount cap |
| Delete data | No | Human approval, soft delete only |

### Data handling, said plainly
Decide before launch what may be sent to a model provider, and enforce it in code rather than in a policy document — redaction at the boundary, an allowlist of fields, and a test proving that a record containing a bank number never leaves. Know the provider’s retention and training terms for the tier you are on, keep them in your vendor file, and re-check them at renewal. Most compliance problems we see are not sophisticated: they are a debug log that captured full request bodies, or a well-meaning feature that pasted an entire customer record into context.

### Test it like an attacker, on a schedule
Put adversarial cases in your evaluation set and run them like any other test: a ticket containing instructions to email an internal document; a document that claims the user is an administrator; a request that would exceed the agent’s remit if honoured. Any run that ends with an action the agent should not have taken is a failing test, not an interesting anecdote. Re-run these after every model version change, because behaviour under adversarial input shifts more between versions than ordinary behaviour does.

Q: Can prompt injection be solved with better prompting?
A: No. Instructions in the system prompt reduce the rate but cannot eliminate it, because the model cannot reliably separate data from instructions. Treat it as an architectural problem: least privilege, content isolation, approval gates and audit logging.

Q: Should the agent use a service account?
A: Only for genuinely public data. For anything user-specific, the end user identity must flow through to the authorisation check, so that the agent can never read or change something the person it is helping could not.

Q: What belongs behind a human gate?
A: Anything you cannot undo, anything visible to a customer, anything above a monetary threshold, and anything the agent is uncertain about. Start with more gates than you think you need and remove them as evaluation numbers justify it — not the other way round.

## Monitoring AI Agents in Production: What to Log and What to Alert On
https://aiagentdevelopment.info/guides/monitoring-ai-agents-in-production
Updated 2026-08-04 · Production and ops

- Trace every run: contexts, tool calls, model versions, stopping reason, later corrections.
- Watch six behavioural metrics; success rate and human intervention rate matter most.
- Alert on rate of change, and attach a representative trace to every alert.
- Read a sample of real runs daily — problems appear in traces before they appear in metrics.

A conventional service is healthy when it responds quickly and does not throw. An agent can do both while being completely wrong — every request answered in two seconds, every one of them citing a policy that was withdrawn in March.

So agent monitoring is a different discipline. It rests on two things: a trace of every run detailed enough to reconstruct what happened, and a small set of behavioural metrics whose movement means something. Everything else is decoration.

### What a useful trace contains
- A run identifier attached to every log line, model call and outbound request.
- The exact context sent to the model at each step, or a hash plus the assembled parts.
- Every tool call with arguments, result, duration and outcome.
- Model name and version per call, plus token counts.
- The stopping reason: completed, step cap, spend cap, human gate, error.
- The final output, and whether a human later corrected or reversed it.

### Six metrics that actually move
| Task success rate | Sustained drop | Model change, data drift, upstream API change |
| Steps per run | Creeping up | Tool errors being retried; degraded retrieval |
| Tool error rate by tool | Spike on one tool | Upstream break — not an agent problem |
| Human intervention rate | Rising | Trust falling, or a new request category |
| Unsupported claim rate | Any rise | Retrieval failing quietly |
| Cost per completed task | Rising with flat volume | Context bloat or extra retries |

### Alert on behaviour, not only on errors
An agent rarely fails loudly. It degrades: slightly more steps, slightly more retries, slightly more escalations, and one morning it is answering from stale documents. Alert on rate of change over a rolling window rather than absolute thresholds — steps per run up 30% week on week is a real signal, while an individual eight-step run is not. Pair every alert with the trace of a representative run, because an alert nobody can investigate quickly becomes an alert everybody mutes.

### Sample and read real runs, every day
No dashboard replaces reading. Pick a handful of runs daily — a couple of successes, every escalation, every capped run — and read them end to end. Every serious problem we have found in a production agent was visible in a trace before it was visible in a metric. Rotate who does the reading; the person who wrote the prompt is the least likely to notice what it is doing wrong.
Add anything surprising you find to the evaluation set the same day. That habit is what turns monitoring into improvement rather than observation.

### Logging without collecting things you should not
Traces contain customer data by construction. Redact identifiers at the point of logging rather than in a later job, set a shorter retention on full contexts than on metrics, and keep tool arguments — which is where personal data concentrates — under the same access controls as the underlying system. Storing everything forever because it might be useful is how a monitoring project becomes a compliance finding.

Q: How long should I keep full traces?
A: Long enough to debug and to satisfy audit needs — commonly 30 to 90 days for full contexts, with metrics and outcome summaries kept much longer. Set the retention deliberately, because these records contain whatever your users typed.

Q: What is the single most valuable alert?
A: A rise in escalations or human corrections. It is the earliest honest signal that behaviour has drifted, and unlike accuracy it needs no labelling — your users and operators are grading the agent for free.

Q: Do I need a specialised observability tool?
A: Not to start. A structured trace table you can query covers most needs. Specialised tools help when you want side-by-side run comparison, evaluation integration and prompt versioning in one place — adopt one when reading traces has become a daily job for several people.

## Cutting AI Agent Costs Without Making It Worse
https://aiagentdevelopment.info/guides/ai-agent-cost-optimization
Updated 2026-08-04 · Production and ops

- Measure cost per completed task, split by task type, before optimising anything.
- Context you no longer need is usually the largest line on the bill.
- Route low-judgement steps down a model tier, one step at a time, with evaluation.
- Cap spend per run and alert on runs that hit the cap.

When a token bill surprises someone, the instinct is to switch to a cheaper model everywhere and accept the quality loss. That is rarely necessary. In the systems we have audited, the majority of spend came from context that did not need to be there and from steps that did not need the expensive model.

The method below is boring and effective: measure first, then apply four changes in order of return, then decide whether you still have a problem. Most teams stop after the second change.

### Measure per run before changing anything
Aggregate monthly spend tells you nothing actionable. Log, for every run: input tokens, output tokens, number of model calls, model used per call, and the task type. Then look at cost per completed task, split by task type. Almost always one or two task types dominate, and within them one step dominates. Optimising anything else is effort spent where the money is not.
Include failed and abandoned runs in the denominator. A retry loop that burns three attempts before succeeding is a cost problem disguised as a quality problem.

### The four changes, in order of return
| Trim context: discard used documents, compress history | 20–40% | Low, if the goal stays pinned |
| Route cheap steps to a smaller model | 20–40% | Low, with per-step evaluation |
| Cache the stable prompt prefix | 10–30% on repeat traffic | Low |
| Reduce steps: better tools, fewer retries | 10–25% | Medium — requires tool work |

### Context is the bill
Every turn resends the accumulated context, so a run of eight steps can pay for the same document eight times. Three habits fix most of it: discard retrieved passages once the step that needed them is finished; compress old turns into short factual notes instead of carrying transcripts; and trim tool results to the fields the agent uses rather than passing whole API responses. None of these reduce capability — they remove text the model was not using anyway, and quality often improves because attention is less diluted.

### Route by step, not by taste
Extraction, classification and formatting rarely need your strongest model; planning and final user-facing prose often do. Move the first group down one tier, run your evaluation set, and keep the change only if the numbers hold. Doing this per step rather than globally is what lets you cut a third of the bill without anyone noticing a difference in the output they read.
- Start with the highest-volume, lowest-judgement step.
- Change one step at a time and re-run the evaluation set after each.
- Log which model produced each decision so a later regression can be attributed.
- Set a per-run spend cap so a pathological case cannot be unbounded.

### What not to do
Do not cut the retrieval that grounds your answers — hallucinated output is far more expensive than tokens once someone has to correct it. Do not remove the critic pass on irreversible actions to save a call. And do not chase micro-optimisations in prompt wording; the savings are noise next to context trimming, and you will spend engineering hours worth more than the difference.

Q: Is caching worth setting up?
A: If your runs share a long stable prefix — system instructions, tool definitions, policy text — yes, and it is one of the cheapest wins available. Structure the prompt so the stable part comes first and the variable part last, or caching cannot help you.

Q: Should I fine-tune to save money?
A: Only for a high-volume, narrow, stable step where a small model with tuning matches a larger one. Fine-tuning adds a maintenance obligation and a re-training cycle; at low volumes routing and context trimming beat it comfortably.

Q: How do I stop one pathological run from costing a fortune?
A: Cap steps and spend per run, detect repeated identical tool calls, and stop with a partial result rather than continuing. Then alert on runs that hit the cap — they are usually a bug worth fixing, not merely an expense.

## Testing AI Agents: Building an Evaluation Set That Earns Its Keep
https://aiagentdevelopment.info/guides/testing-and-evaluating-ai-agents
Updated 2026-08-04 · Production and ops

- Fifty of your own cases beat any public benchmark for deciding whether a change helped.
- Grade outcomes and side effects, never exact transcripts.
- Cover ambiguous inputs, should-refuse cases and empty tool results deliberately.
- Re-run on every prompt, tool, model or retrieval change, and keep sampling live traffic.

The question that separates agents that ship from agents that linger in pilot is simple: how do you know whether yesterday\u2019s change made it better? Without an answer, every prompt edit is a guess, and every regression is discovered by a customer.

An evaluation set is the answer, and it does not require a platform. Fifty cases in a file, a script that runs them, and a grading rule per case will tell you more than any leaderboard, because they are your cases.

### What a case looks like
A case is an input, the starting state of the world, and a checkable expectation. The expectation is almost never an exact string — an agent can be right in several phrasings. Grade the outcome instead: did it call the refund tool with order 4471; did the final answer contain the correct delivery date; did it decline to act and ask a question, as it should have. Where prose quality matters, a rubric graded by a model is acceptable, but keep it to a few explicit criteria, and spot-check the grader by hand from time to time.
Store the state each case needs — the fixture — with the case. A test that only passes on a Tuesday because of live data is not a test.

### Fifty cases, and where they come from
| Common real requests from logs | 20 | Protects the everyday path |
| Known past failures | 10 | Stops regressions returning |
| Ambiguous or under-specified inputs | 8 | Should ask, not guess |
| Should-refuse cases | 6 | Out of scope, unauthorised, unsafe |
| Empty or broken tool results | 6 | The most common real incident |

### Grade outcomes, not paths
Two runs that reach the same correct result by different routes are both correct, and a test suite that insists on one transcript will fail constantly for no reason. Assert on what changed and what was said: the tool calls that had side effects, the final answer’s key facts, whether a human gate was requested. Keep the full trace stored for debugging, but do not assert on it — otherwise every prompt improvement looks like a hundred failures.

### The four numbers to track over time
- Task success rate on the whole set, and separately on the should-refuse subset.
- Unsupported claim rate — answers with facts not present in retrieved evidence.
- Median and 95th-percentile cost and latency per run.
- Human intervention rate: how often a person had to step in, and why.

### Run it in the pipeline, and again after launch
Run the set on every change to prompts, tools, model version or retrieval configuration — those four are the only things that alter behaviour, and all four change more often than people expect. After launch, keep sampling real traffic: pick a few runs a day, grade them by hand, and add anything surprising to the set. An evaluation set that stops growing stops representing your users, usually within a quarter.

Q: How many cases do I need to start?
A: Fifty is enough to catch real regressions and small enough to write in a couple of days. Twenty is enough to begin. The number matters far less than covering the awkward categories — ambiguous inputs, should-refuse cases and empty tool results.

Q: Can I use a model to grade the outputs?
A: Yes, with care. Give it explicit criteria rather than asking whether the answer is good, keep the rubric short, and hand-check a sample regularly. Model graders drift, and a grader that has drifted will happily approve a regression.

Q: Should evaluation block deployment?
A: Block on the safety-critical subset — should-refuse cases and anything involving irreversible actions. For general quality, track the trend and require a human decision on a drop rather than an automatic block, because a small movement can be noise.

## Multi-Agent Systems: When Several Agents Beat One
https://aiagentdevelopment.info/guides/multi-agent-systems-explained
Updated 2026-08-04 · Building agents

- Use several agents only when subtasks are independent, need different tools, and are slow.
- Pass structured objects, not prose, and give the whole request one run identifier.
- Cap cost across the system rather than per agent.
- Evaluate handoffs as well as outcomes, or a regression tells you nothing useful.

Multi-agent diagrams are the most seductive artefact in this field. Boxes with job titles, arrows between them, a coordinator at the top — it looks like an organisation chart, and organisation charts feel like progress.

Then it reaches production and the questions start: which agent produced this wrong number, why did the coordinator accept it, and why does one request now cost eleven model calls. This guide is about the conditions under which the answer is still worth it, and how to build one that stays debuggable.

### The three conditions
Multiple agents pay off when all three hold. The subtasks are genuinely independent — neither needs the other’s output to start. Each needs a different tool set or a different model tier, so specialisation buys something real. And the work is slow enough that doing it in parallel changes the user experience. If only two hold, a single loop with more tools is almost always better, cheaper and easier to fix.
Two agents that must talk to each other repeatedly are one agent with an expensive message bus.

### Topologies, and what each costs
| Supervisor | One agent delegates to specialists | N+1 loops | Supervisor misroutes |
| Pipeline | Fixed handoffs, each stage specialised | Predictable | A stage degrades silently |
| Parallel fan-out | Same task, several perspectives, merged | Highest | Merge step becomes the bottleneck |
| Debate or critic | One proposes, one challenges | 2× per exchange | Agreement without insight |
| Blackboard | Shared state, agents read and write | Unpredictable | Race conditions and loops |

### Design rules that keep it debuggable
- Give every agent a written contract: what it receives, what it returns, and what it must never do.
- Pass structured objects between agents, never free-form prose that the next agent has to re-interpret.
- Give the whole request one run identifier and attach it to every call from every agent.
- Cap the total across the system, not per agent, or costs compound quietly.
- Forbid cycles unless there is an explicit iteration counter and an exit condition.
- Make every agent able to return `I could not do this` and make the coordinator handle it.

### The evaluation problem nobody plans for
With one agent you evaluate outcomes. With several you must also evaluate the handoffs, because a system can produce a wrong answer with every individual agent behaving correctly — the router chose badly, or the merge dropped the important half. Build an evaluation set at both levels: end-to-end outcomes, and per-agent input-output pairs captured from real runs. Without the second, a regression tells you the system got worse and nothing about where.

### A worked example that is worth it
Competitive research is a task that genuinely fits: given ten companies, gather public information on each. The subtasks are independent, each is slow, and the merge is a straightforward aggregation into a table. Ten parallel research agents finish in the time one takes, and a single synthesiser produces the summary. Compare that with a support agent handling one customer question: the steps depend on each other in sequence, so splitting it across agents adds handoffs and latency for nothing at all.

Q: Does a supervisor agent improve accuracy?
A: Only if routing is accurate. A supervisor at 90% in front of specialists at 95% gives you about 85% end to end, and the loss is invisible unless you measure routing separately. Keep the number of specialists small enough to describe each in one sentence.

Q: Is agent debate worth the cost?
A: Sometimes, for genuinely contestable judgements where you can give the critic evidence to check against. For factual lookups it mostly produces agreement, at double the cost. Measure it against a single-pass baseline before adopting it broadly.

Q: How do I debug a multi-agent failure?
A: With a shared run identifier on every call, stored inputs and outputs per agent, and a rendering that shows the handoffs in order. If you cannot reconstruct who said what to whom, you cannot fix it, and you will end up rewriting prompts at random.

## RAG for Agents: Grounding Answers Without Drowning in Context
https://aiagentdevelopment.info/guides/rag-for-ai-agents
Updated 2026-08-04 · Building agents

- Expose retrieval as a tool the agent calls, not a step bolted before every request.
- Chunk on structure, keep chunks self-contained, and attach titles and identifiers.
- Hybrid keyword plus vector retrieval beats either alone on real traffic.
- Require citations and allow an honest empty result, or the agent will invent one.

Retrieval-augmented generation is usually introduced as a pipeline: embed the question, fetch the top chunks, paste them in, generate. That works for a question-answering box. Inside an agent it is the wrong shape, because the agent does not yet know what it needs until it has taken a step.

The version that works treats retrieval as a tool the agent calls when it decides it needs evidence — sometimes twice with different queries, sometimes not at all. That one change removes a great deal of irrelevant context and makes the whole run cheaper and sharper.

### Retrieval as a tool, not a preamble
Expose search as a normal tool with a query argument and a small, structured return: a handful of passages, each with an identifier and a source. The agent decides when to call it, can refine its query after seeing what came back, and can call a different tool instead when the answer is structured data rather than prose. The pipeline version cannot do any of that, and it pays the retrieval cost on every request whether or not the request needed evidence.
Log the queries the agent writes. They are the most honest description you will ever get of what your users are actually asking.

### Chunking decisions that matter more than the embedding model
- Split on structure — headings, sections, list items — not on a fixed character count.
- Keep each chunk self-contained: a chunk that begins with `It also requires` is useless out of context.
- Attach the document title and section heading to every chunk, both for the model and for citation.
- Store an identifier and a URL with each chunk so an answer can point at its source.
- Prefer fewer, larger, meaningful chunks over many small ones; overlap is a patch for bad boundaries, not a strategy.

### Hybrid retrieval beats pure vectors on real corpora
| Conceptual question | Strong | Weak | Vector |
| Exact product code or error string | Weak | Strong | Keyword |
| Rare proper noun | Mixed | Strong | Keyword |
| Paraphrased policy question | Strong | Weak | Vector |
| Most real traffic | Mixed | Mixed | Both, merged and re-ranked |

### Make the agent cite, and make it able to fail
Two requirements do most of the work for trustworthiness. First, every claim drawn from retrieval carries the identifier of the passage it came from, and your interface renders that as a link — this makes unsupported statements visible instead of plausible. Second, the search tool must be able to return nothing, and the agent must be taught that answering `I could not find that in our documentation` is a correct outcome. An agent that cannot fail at retrieval will invent, because inventing is the only remaining option you left it.

### Keeping the index honest
Retrieval quality decays quietly. Documents change, sections are deleted, and the index keeps serving what it last saw. Re-index on a schedule, delete chunks whose source document is gone rather than letting them linger, and keep a small evaluation set of queries with known correct passages so you can measure recall after each change. The failure mode without this is the worst kind: an agent confidently citing a policy that was withdrawn in March.

Q: Should the agent always retrieve before answering?
A: No. Retrieval on every request wastes latency and fills context on questions that do not need evidence. Let the agent call the search tool when it judges evidence is needed, and measure how often it should have and did not.

Q: How many passages should I return?
A: Three to six well-chosen passages beat twenty. More retrieved text dilutes attention, raises cost and increases the chance the agent grounds its answer in a passage that merely looked relevant.

Q: What if retrieval returns nothing useful?
A: That must be a supported outcome. Return an explicit empty result, and instruct the agent to say it could not find the information and offer the next step — asking a clarifying question or handing over to a person.

## Memory in AI Agents: What to Keep, Compress and Throw Away
https://aiagentdevelopment.info/guides/memory-in-ai-agents
Updated 2026-08-04 · Building agents

- Models have no memory; agents have whatever you reassemble into context.
- Four tiers: pinned goal, recent verbatim turns, compressed facts, freshly retrieved data.
- Compress into checkable facts, not narrative, and only on a threshold.
- Persistent memory needs provenance, expiry and a way for the user to correct it.

There is no memory in a language model call. Each turn is a fresh request, and the only thing the model knows is what you assembled into its context this time. Everything people describe as an agent forgetting, or getting worse over a long run, is a decision your code made about what to carry.

Once you accept that, memory design becomes an ordinary engineering problem with a familiar shape: what is always relevant, what is recently relevant, what can be summarised, and what should be fetched fresh instead of stored at all.

### The four tiers
| Pinned | Goal, constraints, user identity, policy | Never | 200–500 tokens |
| Recent | Last few turns verbatim, with tool results | Rolling window | 2–5 turns |
| Compressed | Older turns as short factual notes | Rewritten as it grows | Under 500 tokens |
| Retrieved | Documents and records fetched for this step | Discarded after use | Per step |

### Compress facts, not prose
The usual mistake is to summarise old turns as narrative — the user asked about their order and the agent looked it up. That reads well and helps nothing. Compress into the facts a later step might need: order 4471, status shipped, customer requested refund, refund policy allows 30 days, no refund issued yet. Structured, checkable, and a tenth of the size. If a later step is going to make a decision, the compression must preserve the inputs to that decision or it has failed at its only job.
Compress on a threshold, not every turn. Re-summarising a summary repeatedly is how details quietly disappear.

### Retrieval is not memory
Documents pulled from a knowledge base belong to the step that needed them. Keeping them in the running context after that step is finished is the fastest way to a bloated, expensive, distracted run. Fetch, use, cite, discard — and if a later step needs the same fact, fetch it again. Retrieval is cheap; a context full of stale documents is not.

### Memory that persists between sessions
Long-lived agents accumulate genuinely useful facts about a user or account: preferences, prior decisions, constraints that will not change. Store these deliberately, in a small structured record with an explicit write step, rather than by letting conversation history pile up. Three rules keep it healthy: write only facts a future run would act on, always record where a fact came from, and give every fact an expiry or a review date. Persistent memory without provenance and expiry becomes a slowly rotting source of confident errors.
- Write on purpose — a tool call, not a side effect of chatting.
- Store the source and the date alongside every fact.
- Cap the record size and expire what has not been used.
- Let the user see and correct what is stored about them.

### Symptoms and their causes
| Forgets a constraint from early in the run | Constraint was not pinned; it was trimmed with the history |
| Quality degrades after several steps | Context is diluted with stale tool output |
| Repeats a completed step | Result was summarised away without an outcome marker |
| Cost climbs with run length | Retrieved documents are accumulating instead of being discarded |
| Confidently states something outdated | Persistent memory has no expiry or provenance |

Q: How much history should I keep verbatim?
A: Three to five turns covers most reasoning without dominating the context. Keep tool results attached to their calls, and compress anything older into structured facts rather than dropping it silently.

Q: Should I use a vector database for agent memory?
A: For retrieving documents, often yes. For the running state of a single run, no — that is a small structured object in your own store. Conflating the two produces both a fuzzy state machine and an unfocused search index.

Q: How do I stop persistent memory from going stale?
A: Give every stored fact a source, a date and an expiry, and make the agent prefer freshly retrieved data when both exist. Expose the stored record to the user so wrong facts can be corrected rather than silently repeated.

## Tool Calling: How to Design Tools an Agent Uses Correctly
https://aiagentdevelopment.info/guides/tool-calling-for-ai-agents
Updated 2026-08-04 · Building agents

- Most agent failures are tool design failures, not prompt failures.
- One tool one job; use types, not prose, to constrain arguments.
- Write errors as short instructions; treat empty results as legitimate outcomes.
- Validate every argument and resolve identifiers against what this user may see.

When an agent behaves badly, the instinct is to rewrite the prompt. In our experience the prompt is the cause perhaps a third of the time; the rest of the time the tools were designed for a program, not for a reader who has to infer from names and descriptions what a function does.

Tools are the agent\u2019s entire ability to affect the world, and their definitions are literally part of the model\u2019s context. Designing them well is cheaper and far more durable than prompt tuning, because a good tool constrains behaviour instead of requesting it.

### Seven rules that prevent most bad calls
- One tool, one job. `search_orders` and `refund_order` beat a single `manage_order` with a mode argument.
- Types over prose. Enums, integer ranges and formats do work a description never will.
- Names that say what happens. `send_email_to_customer` is unambiguous in a way `notify` is not.
- Errors as instructions: state what was wrong and what to do next, in one short sentence.
- Empty results are results. Returning an explicit no-match beats an exception the model treats as retryable.
- Idempotency keys on anything with a side effect, so a retry cannot duplicate it.
- Small returns. Trim payloads to the fields the agent needs; a 40 KB JSON blob buys confusion, not context.

### Before and after
| `query(sql)` | Unbounded power, unauditable | `get_orders_by_customer(customer_id, limit)` |
| `date: string` | Model invents formats | `date: string, format YYYY-MM-DD` |
| `HTTP 500` | No action implied, retried forever | `Order service unavailable. Tell the user to try later.` |
| Returns full record | Fills context, dilutes attention | Returns six named fields |
| `update_status(id, status)` | Any status, any record | `cancel_order(id)` with permission check |

### Descriptions are prompt, so write them like prompt
The description field is not documentation for your colleagues; it is text the model reads while deciding. Say when to use the tool and when not to, name the one precondition that matters, and give one example argument. Three sentences beats three paragraphs — long descriptions crowd out the rest of the context and rarely change behaviour. And review them together, in one file: tools that make sense individually often overlap in ways that only become visible when they are read as a set.
If two tools could plausibly answer the same request, the agent will sometimes pick the wrong one. Either merge them or make the boundary explicit in both descriptions.

### Validate before you execute, always
Never pass model output to a system call unchecked. Validate the arguments against the schema, resolve identifiers against records the current end user is allowed to see, and reject anything that does not match rather than coercing it into something plausible. A rejection with a clear message is a good outcome: the agent learns the constraint and tries something else within the same run. A silent coercion is how an agent updates the wrong record and nobody finds out until the report looks strange.

### Test tools separately from the agent
Give each tool its own tests: valid call, invalid arguments, permission denied, empty result, upstream timeout. Then test the agent against a stub tool layer so you can force those conditions deliberately. Almost every production incident we have reviewed reproduces trivially at this level once someone thinks to try — the empty-result case in particular, which is rare in development and routine on a real Tuesday afternoon.

Q: How many tools is too many?
A: Beyond roughly ten in one loop, selection accuracy starts to fall and descriptions crowd the context. If you need more, group them behind a narrow router or split into specialised agents with small tool sets each.

Q: Should tools return raw API responses?
A: No. Return a small, stable shape with the fields the agent actually needs. Raw responses waste context, expose fields the agent may misuse, and couple your prompt behaviour to somebody else’s API version.

Q: How do I stop the agent inventing arguments?
A: Constrain them: enums instead of free strings, explicit formats, and identifiers that must resolve against real records. Then validate and return a clear rejection. Invented arguments are usually a sign the tool asked for something the agent had no way to know.

## AI Agent Architecture: The Patterns That Hold Up in Production
https://aiagentdevelopment.info/guides/ai-agent-architecture-patterns
Updated 2026-08-04 · Building agents

- Default to a bounded loop with explicit stopping conditions and a linear trace.
- A tool gateway — validation, authorisation, rate limits, audit — is the highest-value component.
- Planner–executor buys visible intent; a critic pass buys fewer unsupported claims.
- Tier your memory: goal, recent turns, compressed history, freshly retrieved knowledge.

Agent architecture discussions usually start at the wrong end, with a diagram of boxes named after concepts. The useful version starts with the failure you are trying to prevent, because every pattern below exists to stop one specific bad afternoon.

These six are the ones we keep reaching for. They compose: a production agent is typically a bounded loop with a tool gateway, two memory tiers and a human gate, and a critic pass only where the cost of a wrong answer justified another model call.

### 1. The bounded loop
The base case, and the one you should default to. A single loop over a small tool set with explicit stopping conditions: a step cap, a spend cap, repeat detection and a wall-clock limit. Its virtue is a linear trace an engineer can read from top to bottom. Every other pattern here is an addition to this, not a replacement for it.
If you cannot draw your agent as a loop with a list of exits, you do not yet have an architecture; you have a prompt with ambitions.

### 2. Planner–executor with re-planning
For tasks that reliably take more than about five steps, ask for an explicit numbered plan first, execute the steps, and re-plan when a step fails rather than continuing down a stale plan. The benefit is not accuracy — it is that a human can see what the agent intends before it acts, which makes both approval and debugging tractable. The cost is one extra model call, plus the discipline of treating the plan as revisable rather than sacred.

### 3. The critic pass
A second model call reviews the draft answer or the proposed action against the goal and the retrieved evidence, and can send it back once. This catches a meaningful share of confident-but-unsupported output. Use it where a wrong result is expensive and one extra call is not: outbound customer messages, financial summaries, code changes. Do not use it everywhere — it doubles cost and latency, and on easy tasks it mostly agrees with itself.
| Bounded loop | 0 | Traceability, cost control | Never — this is the base |
| Planner–executor | 1–2 | Visible intent, auditability | Tasks under five steps |
| Critic pass | 1 per checked output | Fewer unsupported claims | Cheap, reversible outputs |
| Tool gateway | 0 | Permissions, audit, rate limits | Prototypes only |
| Memory tiers | 0–1 | Relevance at long context | Short single-turn runs |
| Human gate | 0 | Irreversible actions stay safe | Nothing irreversible exists |

### 4. The tool gateway
Do not let the agent call your systems directly. Put one layer in front of every tool that does four things: validates arguments against a schema, checks that this end user may touch this record, applies a rate limit, and writes an audit line with the run identifier. This is the single highest-value piece of infrastructure in an agent system, and it is ordinary code that takes days rather than weeks. It also means switching agent frameworks never touches your security posture.

### 5. Memory in tiers
One undifferentiated conversation history is the most common cause of an agent that gets worse as a run continues. Separate what you carry: the goal and constraints, which never get trimmed; the recent turns, kept verbatim; the older turns, compressed into a short factual summary; and retrieved knowledge, fetched fresh per step and never accumulated. Long contexts are expensive and attention is finite — carrying everything is not thoroughness, it is dilution.

### 6. The human gate
Every irreversible action sits behind an explicit approval step with enough context for a person to decide in seconds: what will happen, to which record, why the agent believes it should, and what it will do if declined. The gate is a product feature, not a limitation — it is what lets you ship an agent into a system where mistakes are expensive, and it is what you remove selectively once your evaluation numbers earn it.

Q: Do I need all six patterns?
A: No. Start with the bounded loop and the tool gateway; those two are close to mandatory for anything touching real systems. Add the human gate the moment an irreversible action appears. The rest are earned by specific failures you can point to in a trace.

Q: Does a critic pass really improve accuracy?
A: On tasks where the model can produce a plausible but unsupported answer, meaningfully yes — particularly when the critic is given the retrieved evidence and asked to check claims against it. On simple lookups it mostly agrees with the first answer and doubles your cost.

Q: Where should the tool gateway live?
A: In your own service, between the agent and your systems, with the end user identity flowing through it. If it lives inside the agent framework, you will re-implement it the next time you change frameworks, and your security review will start again from zero.

## No-Code Agent Platforms vs Custom Builds: An Honest Comparison
https://aiagentdevelopment.info/guides/no-code-vs-custom-ai-agents
Updated 2026-08-04 · Frameworks and models

- Platform and custom are phases, not rivals; the error is staying too long in one.
- Per-user permissions, product ownership and volume push you to custom.
- Prove the workflow on a platform, then rebuild only the parts that earned it.
- Export prompts, tool definitions and logs from day one.

The no-code versus custom argument is usually conducted between people with something to sell. Having built both, our view is duller and more useful: they are phases, not rivals, and the mistake is staying in one longer than the evidence supports.

A platform is the cheapest way to discover what your task actually requires. A custom build is how you get control of permissions, unit cost and the product surface once you know. Below is how to tell which phase you are in.

### Side by side, without the marketing
| Time to first working version | Days | Weeks |
| Cost shape | Per seat or per run, ongoing | Engineering up front, then infrastructure |
| Access to internal systems | Whatever connectors exist | Anything you can write code against |
| Permissions per end user | Usually coarse | As fine-grained as you build |
| Evaluation and regression testing | Vendor-provided, sometimes shallow | Yours, as deep as you invest |
| Portability | Configuration lives in the vendor | Repository you own |
| Right when | Proving value, standard tasks, small team | Product surface, real permissions, volume |

### Four questions that settle it quickly
- Does the agent need per-user permissions on internal data? If yes, custom, almost always.
- Is the agent part of what you sell? If yes, custom — you cannot outsource your product surface.
- Will you run more than a few thousand tasks a month? If yes, do the arithmetic on per-run pricing before committing.
- Do you need your own evaluation set and audit trail? If yes, check what the platform exports before, not after.

### The hybrid pattern that works
Prove the workflow on a platform, instrument everything, and let it run for a month with real users. You will learn three things you could not have designed for: which requests actually arrive, which tools get used, and where humans intervene. Then rebuild only the parts that earned it — usually the two tools that touch sensitive systems and the evaluation harness — and leave the rest where it is. Rebuilding everything at once discards the operational knowledge you just paid to acquire.
Export your prompts, tool definitions and conversation logs from day one. If a platform makes that hard, treat it as a finding about the platform.

### What custom actually costs, so nobody is surprised
A custom build is not just the agent loop. It is the tool layer with typed arguments and error contracts, the permission checks, the evaluation set, tracing you can read, a deployment path, and an owner when the model provider deprecates a version. That is where our estimate of two to four months for a customer-facing agent comes from, and why an internal agent behind a narrow API is much cheaper. Teams that budget only for the loop deliver a demo and stall.

### Signals it is time to leave the platform
- You are writing workarounds for a connector rather than features.
- Per-run cost has become a line item somebody asks about in a meeting.
- A security review has blocked the next step and the platform cannot answer the question.
- You want to change one behaviour and cannot express it in the builder.
- The agent is now part of the customer experience and you have no way to test it properly.

Q: Can a no-code platform be the permanent answer?
A: Yes, for internal, standard, moderate-volume tasks where the cost of a wrong answer is low. Plenty of useful automation should never become a code project — the test is whether permissions, volume or product ownership ever force the question.

Q: Is a custom build always more accurate?
A: No. Accuracy comes from tool design, grounding and evaluation, all of which you can do on a platform. Custom gives you control and economics, not intelligence.

Q: What is the biggest hidden cost of custom?
A: Maintenance. Models get deprecated, APIs change, and your evaluation set needs re-running against each change. Budget roughly 15–25% of the build cost per year, and give the agent a named owner the way you would any production service.

## The Model Context Protocol, Explained for Builders
https://aiagentdevelopment.info/guides/model-context-protocol-explained
Updated 2026-08-04 · Frameworks and models

- MCP standardises tool discovery and invocation between agent clients and servers.
- It does not handle authentication, authorisation or approval — those stay yours.
- Wrap narrow capabilities and enforce permissions inside the server, per call.
- Third-party servers are dependencies whose descriptions enter your model context.

Every team that builds more than one agent writes the same adapter twice: connect to a system, describe what it can do, expose those capabilities to a model in whatever shape today\u2019s client expects. The Model Context Protocol exists to stop that duplication by standardising the interface between an agent client and a tool server.

That is genuinely useful, and it is also narrower than the enthusiasm suggests. MCP describes how capabilities are advertised and invoked. It does not decide who is allowed to invoke them, and confusing those two is where the security incidents come from.

### What the protocol standardises
- Discovery: a server tells a client which tools and resources it offers, with schemas.
- Invocation: the client calls a tool with typed arguments and receives a structured result.
- Resources: read-only content the client can pull into context on request.
- Transport: a common wire format so a client and a server written by different people interoperate.

### What it deliberately does not do
MCP does not authenticate your users, does not decide which records a given user may read, and does not decide whether an action needs human approval. Those remain your responsibility, and they must live on the server side of the boundary — a client asking nicely for permission is not a permission system. The most common architectural error is to expose a broad tool such as `run_query` over MCP and rely on the agent’s prompt to keep it in bounds.
Treat every MCP tool as if a confused or manipulated caller will invoke it with the worst plausible arguments, because sooner or later one will.

### Where it pays off today
| One internal system, several agent clients | High — write the server once |
| Desktop assistants reading local context | High — the ecosystem is built around this |
| A single agent with three bespoke tools | Low — direct function calls are simpler |
| Third-party tools you do not control | Medium — convenient, but audit the server |

### A safe way to adopt it
- Wrap narrow capabilities, not general power: `get_order(id)` rather than `sql(query)`.
- Enforce authorisation inside the server, per call, using the identity of the end user rather than a service account with everything.
- Return honest, short errors — `not found`, `not permitted` — so the agent can respond sensibly instead of retrying.
- Log every invocation with arguments and caller identity; this is your audit trail when someone asks what happened.
- Pin the servers you depend on to versions you have reviewed, exactly as you would any dependency.

### The supply-chain question
A third-party MCP server is code that describes tools to your agent and receives whatever arguments your agent decides to send. Tool descriptions are part of the model’s context, which means a malicious or careless description can influence behaviour. Review servers before adopting them, prefer ones you can read, and keep untrusted servers away from any client that also has access to sensitive systems. This is ordinary dependency hygiene applied to a new kind of dependency.

Q: Do I need MCP to build an agent?
A: No. Direct function calls are simpler for a single agent with a handful of bespoke tools. MCP starts paying when the same capability must be reachable from several clients, or when you want to consume tools built by other teams.

Q: Is MCP secure by default?
A: It is a transport and discovery standard, not a security model. Authentication, per-user authorisation and approval gates are yours to implement on the server side, and they must not be delegated to the agent’s prompt.

Q: Can MCP servers be a prompt injection vector?
A: Yes — both through tool descriptions that enter the model context and through returned content. Treat server output as untrusted input, keep tool scopes narrow, and do not point an agent with production write access at servers you have not reviewed.

## Choosing a Model for Your Agent: Capability, Latency and Cost
https://aiagentdevelopment.info/guides/choosing-an-llm-for-your-agent
Updated 2026-08-04 · Frameworks and models

- Choose a model per step, not one model for the whole agent.
- Benchmarks make the shortlist; thirty of your own cases make the decision.
- Latency, structured-output reliability and real context length are the binding constraints.
- Pin explicit versions and keep the eval set one command away from running.

The question teams ask is which model is best for agents. The question that produces a good system is which model is best for this step, on our data, at our latency budget — and the answer is usually more than one model.

An agent run is not homogeneous. Deciding the next action needs reasoning. Extracting three fields from a returned document does not. Summarising a result for a user does not. Treating all of these as one procurement decision is how teams end up paying frontier prices to reformat JSON.

### Split the run before you choose
| Plan or choose next action | Reasoning, instruction adherence | Strongest you can afford |
| Call a tool with arguments | Reliable structured output | Mid tier with strict schemas |
| Extract fields from a result | Accuracy on short text | Small and fast |
| Classify or route | Consistency | Small, or a fine-tuned classifier |
| Write the user-facing reply | Tone and clarity | Mid tier |

### Benchmarks are a shortlist, not a decision
Public benchmarks tell you which models are plausible. They cannot tell you which one handles your tool schemas, your document formats and your awkward customers, because none of those are in the benchmark. Build a set of thirty real cases from your own logs — including the five that embarrass you — and run the shortlist against it. The gap between benchmark order and your-own-data order is routinely large enough to change the decision.
Include cases where the correct behaviour is to refuse or to ask a question. Models differ far more on knowing when to stop than on knowing what to say.

### The three constraints that actually bind
- Latency floor: each model call has one, and an agent makes several. Measure the whole run, not one call.
- Structured output reliability: a model that returns valid arguments 97% of the time fails one run in ten with three tool calls.
- Context handling: long contexts cost more and degrade attention; measure accuracy at the context length you will really use, not at the marketing maximum.

### Routing without a research project
Model routing sounds sophisticated and is usually a config file. Assign a default model per step type, allow an override per tool, and log which model produced each decision so you can attribute failures. Start by moving only the extraction and classification steps down a tier; that alone often removes a third to a half of the token bill without touching the quality of the part users notice.

### Plan for deprecation, because it is coming
Model versions are retired on the provider’s schedule, not yours. Two habits make that a non-event: pin an explicit version rather than a floating alias so nothing changes under you silently, and keep your evaluation set runnable with a single command so re-qualifying a replacement is an afternoon rather than a project. Teams without those habits discover the deprecation email and the incident on the same morning.

Q: Should I use the biggest model for everything?
A: Only if you have not measured. The decision step usually benefits from the strongest model; extraction, classification and formatting rarely do. Splitting by step is the single easiest cost reduction available and it does not touch quality where users see it.

Q: Do open-weight models work for agents?
A: For narrow, well-defined steps with strict schemas, frequently yes, and the economics at volume can be compelling. For open-ended planning across many tools they still tend to need more scaffolding. Test on your own thirty cases rather than on a leaderboard.

Q: How often should I re-evaluate my model choice?
A: Whenever a provider ships a version you might adopt, and on a fixed cadence of about two quarters otherwise. That is only sustainable if re-running your evaluation set is a single command, which is the real reason to invest in it.

## Agent Orchestration: When You Need It and When It Is Overhead
https://aiagentdevelopment.info/guides/agent-orchestration-libraries-compared
Updated 2026-08-04 · Frameworks and models

- Orchestration buys durability, idempotency, branching and resumption.
- Ask what a half-failed run costs to repeat; that answer decides adoption.
- A queue, a state row and idempotency keys deliver most of the benefit cheaply.
- Keep prompts and tool schemas outside workflow definitions whatever you choose.

Orchestration libraries solve a genuine problem: a run that takes minutes, touches several systems, and must survive a process restart without repeating the payment it already made. That problem is real and unpleasant to solve by hand.

It is also not the problem most agents have. A support agent that answers in fifteen seconds and can safely be retried from scratch needs none of it. This guide separates the cases, so you adopt orchestration when the failure modes justify it rather than when the architecture diagram looks lonely.

### What orchestration actually gives you
- Durable state: the run survives a deploy, a crash or a scale-down.
- Idempotent steps: a retried step does not repeat a side effect that already happened.
- Branching and joins: real control flow, not a prompt describing control flow.
- Resumption: pause for a human approval that arrives four hours later.
- Observability by construction: each step is a first-class object with a status.

### The test that decides it
Ask one question: if this run died halfway, what would it cost to start it over? If the answer is a few cents and a few seconds, start it over — you do not need durability, you need a retry. If the answer is a duplicated refund, a second email to a customer, or twenty minutes of a person’s waiting, you need durable, idempotent steps and you should stop hand-rolling them.
Most teams discover their real answer the first time a deploy lands mid-run. It is cheaper to decide before that.

### Where the complexity shows up
| Local development | Run the file | Run the worker and the state store too |
| Debugging | Read one linear trace | Correlate steps across a run history |
| Deploying mid-run | Run dies | Run resumes |
| Human approval steps | Awkward; usually a new request | First-class pause and resume |
| Cost of a bug in step 3 | Rerun everything | Rerun step 3 |

### A middle path most teams miss
You do not have to choose between a bare loop and a full orchestration platform. A modest queue, a state row per run, and idempotency keys on the two side-effecting tools cover perhaps eighty percent of the benefit for a fraction of the operational surface. Write the run identifier and step index into every external call you make; make the two dangerous tools reject a repeated identifier. That is an afternoon of work and it removes the failure mode people actually get burned by.

### If you do adopt one
- Keep agent logic — prompts, tool schemas, stopping rules — outside the workflow definitions.
- Make every step idempotent even though the framework promises exactly-once; promises meet networks eventually.
- Cap total run cost at the orchestration layer, not only inside the agent loop.
- Export traces in a format you can read without the vendor console, because incidents happen at inconvenient times.

Q: Can I use orchestration for a simple chat-style agent?
A: You can, and it will work, but you will pay for it in local development friction and debugging indirection every day for a benefit you claim rarely. Reach for it when runs are long, expensive to repeat, or must pause for humans.

Q: Is a message queue enough?
A: Often, yes. A queue plus a per-run state row plus idempotency keys on side-effecting calls covers the common failure modes. Move up when you need real branching, joins, or pauses measured in hours.

Q: How do I keep runs debuggable once steps are distributed?
A: Give every run a stable identifier, attach it to every log line, model call and outbound request, and store the exact context sent to the model at each step. Distributed systems are debuggable when correlation is designed in and miserable when it is added later.

## Choosing an AI Agent Framework: What Actually Matters
https://aiagentdevelopment.info/guides/best-ai-agent-frameworks
Updated 2026-08-04 · Frameworks and models

- Framework rankings age fast; the questions that decide fit do not.
- Three bargains: provider SDK, orchestration library, managed platform.
- Keep prompts, tool schemas, eval set and trace format in your own repo.
- Build the same small agent twice before committing — measure debuggability, not accuracy.

Any article that ranks agent frameworks by name is out of date before it is indexed. Libraries in this space rewrite their core abstractions every couple of releases, and the one that looks best in a benchmark today may have moved on by the time your project ships.

So this guide does something more durable: it lists the eight questions that actually determine whether you will still be glad about your choice in six months, and explains what each answer costs you. Take it to whatever the current shortlist is and you will get a decision you can defend.

### The eight questions, in the order they matter
- Can I read the loop? If you cannot find the file where the model output is turned into a tool call, you cannot debug a bad run.
- What happens on a tool failure — does it surface to me, or is it retried invisibly with a different prompt?
- Is my prompt the framework’s prompt? Hidden system text you did not write will surprise you in an audit.
- Can state be persisted and resumed, or does a crash lose the run?
- How are tools defined, and can I reuse those definitions outside this framework?
- What is the upgrade story — have the core abstractions been renamed in the last two releases?
- Can I swap models without swapping frameworks?
- What does it add to a cold start and to each turn?

### Three broad categories, three different bargains
| Provider SDK plus your own loop | Total visibility, minimal dependencies | You write retries, state, persistence yourself | One agent, few tools, high debuggability need |
| Orchestration library | Durable state, branching, retries, resumption | Some visibility; upgrade churn | Long-running or multi-step workflows |
| Managed agent platform | Hosting, tracing, evaluation, UI | Portability; per-seat or per-run pricing | Small teams, standard tasks, fast proof |

### Write the parts that are yours to keep
Whatever you choose, four assets should live in your own repository in a form no framework owns: the prompts, the tool definitions and their JSON schemas, the evaluation set, and the trace format. These are the things that took real work to get right. If they are expressed as plain data and thin adapters, changing framework is a day of adapter work. If they are expressed as framework decorators and inherited classes, changing framework is a rewrite — and you will therefore not change, even when you should.

### The evaluation nobody does but everybody should
Before committing, build the same small agent twice: once on your shortlist favourite and once on the provider SDK with a hand-written loop. Give both the same three tools and the same ten test cases. You are not measuring accuracy — both will be similar. You are measuring how long it took, how readable the trace is, and how easy it was to find out why case seven failed. That afternoon has saved every team we know of far more than it cost.
Keep the hand-written version. It becomes your reference implementation when you need to prove whether an oddity comes from your prompt or from the framework.

### Signals that you have outgrown your choice
- You are reading the framework source more often than your own.
- You maintain a patch or a fork to get behaviour you needed.
- Upgrades are deferred because of breaking renames, and you are two majors behind.
- Half your prompt exists to counteract text the framework injects.
- Tracing requires a bespoke exporter because the built-in one hides tool arguments.

Q: Do I need a framework at all for a first agent?
A: No. A first agent with three tools is a loop, a schema list and a stopping condition. Building it by hand once teaches you what a framework would be doing on your behalf, which makes the later choice much better informed.

Q: Is a managed platform a trap?
A: Not if you keep your prompts, tool schemas and evaluation set portable. Platforms are genuinely fast to a working result. The risk is not the platform, it is letting your intellectual property exist only as configuration inside it.

Q: How much does framework choice affect accuracy?
A: Far less than people expect. Accuracy comes from tool design, grounding and evaluation. Frameworks affect development speed, debuggability and operational features — which matter, but not in the way benchmark tables imply.

## When Not to Use an AI Agent (and What to Build Instead)
https://aiagentdevelopment.info/guides/when-not-to-use-an-ai-agent
Updated 2026-08-04 · Agent basics

- Fixed sequences want a pipeline with a model step, not an agent.
- Arithmetic, exact matching and sub-second latency are all wrong fits.
- No evaluation set means no way to know if a change helped — build twenty examples first.
- Irreversible high-value actions belong behind a human gate, with the agent drafting.

We build agents for a living, which is exactly why this page exists. The fastest way to damage a team's confidence in this technology is to put an agent on a task that never needed one, watch it be 94% right where a script was 100% right, and spend the next quarter defending it.

Below are the six situations where we say no, and the thing we suggest instead. None of them is a statement about model capability; they are statements about where nondeterminism is a cost rather than a feature.

### 1. The steps never change
If the sequence is fixed — pull file, validate columns, transform, load, notify — you do not need anything to decide what happens next, because nothing decides. Write the pipeline. If one step needs judgement, such as classifying a free-text field, call a model for that one step and keep the rest deterministic. You get the model where it helps and predictable behaviour everywhere else.
This is the single most common overbuild we see. A model call inside a pipeline is not a lesser thing than an agent; it is the correct thing.

### 2. The task is arithmetic or exact matching
Totals, reconciliations, tax, eligibility rules with published thresholds: these have right answers and existing implementations. A model can explain a calculation beautifully and still get it wrong occasionally, and occasionally is a catastrophe in finance. Compute in code, then let the model do what it is genuinely better at — explaining the result to a person in their own language.

### 3. Latency budget under a second
An agent that plans, calls two tools and answers cannot reliably do it in under a second, because each model call has its own latency floor. If you are inside a checkout, a search-as-you-type box, or a call routing decision, either move the work off the critical path and show the result later, or use a classifier and a lookup. Users forgive a slow answer they asked for; they do not forgive a slow page.

### 4. Nobody can say what a correct run looks like
If the team cannot produce twenty examples of the task done right, you have no evaluation set — and without an evaluation set you have no way to know whether a change helped. Build the examples first. Often the act of writing them reveals that the task is really three different tasks, two of which are trivial and one of which is the real problem.
Twenty labelled examples is a low bar deliberately. If a team cannot reach it, the task is not yet understood well enough to automate by any method.

### 5. Every action is irreversible and high value
Wire transfers, contract signatures, production deletions. You can absolutely put an agent in front of these — as a drafter that assembles the case and hands it to a human. What you should not do is give an autonomous loop unattended write access to something that cannot be undone, on the strength of an accuracy figure from a test set that did not contain your worst week.

### 6. The data it needs is not accessible
An agent is only as capable as its tools, and its tools are only as capable as your APIs. If the information lives in a system with no read API, or in a spreadsheet that three people edit by hand, the agent will be reduced to guessing. Fix the access first. That work is unglamorous and it is also the majority of the value: teams that build the read API often find the agent afterwards is a two-week job.

Q: So when is an agent clearly the right tool?
A: When the next step genuinely depends on what the last step returned, when several tools might be needed in an order you cannot fix in advance, and when a human currently does this by looking things up and deciding. That combination is where the loop earns its cost.

Q: We already built an agent for a fixed pipeline. Should we rip it out?
A: Not necessarily — measure first. If it is reliable and the cost is acceptable, leave it and spend your effort elsewhere. Replace it when you can point to a specific pain: unpredictable latency, cost per run, or failures nobody can reproduce.

Q: Can an agent be part of an otherwise deterministic system?
A: Yes, and that is often the best design. Keep the spine deterministic and give the agent one bounded region where judgement is required, with a clear contract on what it may return. You get flexibility where you need it and predictability everywhere else.

## Types of AI Agents: Five Shapes That Cover Almost Everything
https://aiagentdevelopment.info/guides/types-of-ai-agents
Updated 2026-07-28 · Agent basics

- Five practical shapes: tool-augmented responder, single loop, planner–executor, router, collaborating agents.
- Each rung up buys capability by spending traceability and cost.
- Most production agents are a single loop with three to six tools.
- Move up only with a trace proving the simpler shape structurally cannot work.

Academic taxonomies of agents — reflex, model-based, goal-based, utility-based — are useful for exams and nearly useless when you are choosing what to build on Monday. What matters in practice is the shape of the control flow, because that determines your cost, your latency and how hard the thing is to debug.

Five shapes cover almost every agent we have shipped or reviewed. They form a rough ladder of complexity, and the most common expensive mistake is starting two rungs higher than the task requires.

### The five shapes, from cheapest to hardest
| Tool-augmented responder | One model call, maybe one tool | Lookups, enrichment, classification | Barely an agent; fine |
| Single-loop agent | Model loops over a small tool set | Support tasks, research, triage | Wandering on long tasks |
| Planner–executor | Plan once, execute steps, re-plan on failure | Multi-step ops, migrations | Stale plans after step three |
| Router with specialists | One router picks a narrow sub-agent | Broad domains with distinct skills | Routing errors compound |
| Collaborating agents | Several agents exchange results | Genuinely parallel research or review | Cost, latency, untraceable failures |

### Start one rung lower than feels right
The single-loop agent solves far more real problems than its reputation suggests, and it has one enormous advantage: a linear trace that a human can read top to bottom. Every rung above it buys capability by spending traceability. Before you move up, be able to name the specific case that the simpler shape failed on, with a trace to prove it. Teams that skip this step end up with a five-agent system whose bugs nobody can localise, doing a job a loop with four tools was already doing at a fifth of the cost.

### How to tell which rung you actually need
- If the task is one lookup and one decision, you need a tool-augmented responder.
- If the tools are few and the order varies, you need a single loop.
- If a human doing this job would write a checklist first, you need planner–executor.
- If the job splits cleanly into distinct expertises with different tools, you need a router.
- If two subtasks genuinely have no dependency and both are slow, collaboration may pay for itself.

### The specialist trap
Routers look tidy on a diagram and behave badly at the edges. The router sees only the request, not what the specialists would have found, so it must guess — and a wrong guess sends the request to a specialist that cannot say anything useful. Two mitigations help: let a specialist return `not mine` and route again once, and keep the number of specialists small enough that the router prompt can describe each in one clear sentence. If you cannot describe the boundary in a sentence, neither can the router.
Measure routing accuracy separately from task accuracy. A 90% accurate router in front of 95% accurate specialists gives you 85% end to end, and the diagnosis is invisible if you only track the total.

### Shape and cost, honestly
Cost grows faster than the diagram suggests. A single-loop agent on a typical support task costs a handful of model calls. A planner–executor adds one planning call and often a re-plan. A router adds a call before anything useful happens. Collaborating agents multiply: three specialists each running their own loop is three loops, and if a coordinator reviews their output that is a fourth. None of that is a reason to avoid the higher shapes; it is a reason to reach them deliberately, with a number in front of you.

Q: Are multi-agent systems better than a single agent?
A: Only when the subtasks are genuinely independent and each needs different tools or a different model. Otherwise you have paid for extra latency, extra tokens and a failure surface that is much harder to trace, in exchange for a diagram that looks impressive in a slide deck.

Q: What is the most common shape in production?
A: The single-loop agent with three to six tools and a human gate on irreversible actions. It is unglamorous, it fits most real tasks, and its linear trace means an engineer can diagnose a bad run without special tooling.

Q: How do I know when to move up a rung?
A: When you have a trace of a real failure that the simpler shape structurally cannot fix — not a case it got wrong once, but a class of case it cannot represent. Write that trace down; it is also the test case that proves the new shape helped.

## How AI Agents Work: The Loop, Step by Step
https://aiagentdevelopment.info/guides/how-ai-agents-work
Updated 2026-07-28 · Agent basics

- An agent turn is: assemble context, decide, validate, execute, record, check stopping conditions.
- The model sees only what you put back into context — trimming decisions cause most odd behaviour.
- Write tool errors as instructions the model can act on, not as diagnostics.
- Traces are the primary debugging tool; build them before the second feature.

Agents look like magic in demos and like plumbing in production. The reason is that the interesting part is not the model output but the loop that consumes it, and the loop is short enough to read in one sitting.

This guide walks a single request through that loop end to end: what the model sees on each turn, what your code does with the result, how a failing tool comes back, and what makes the loop stop. If you can narrate this for your own system, you can debug it. If you cannot, no amount of prompt tuning will make it reliable.

### One turn of the loop, in order
- Assemble context: the goal, the tool definitions, relevant retrieved facts, and a trimmed history of what has already happened.
- Ask the model for the next step. It either answers directly or requests a tool call with arguments.
- Validate the arguments before doing anything — types, ranges, and whether this caller is allowed to touch this record.
- Execute the tool. Catch failures and turn them into short, factual messages rather than stack traces.
- Append the call and its result to the history, then check the stopping conditions.
- Repeat, or return the final answer with whatever the agent actually did.

### What the model can and cannot see
The model has no memory of the previous turn beyond what you put back into the context. That single fact explains most confusing agent behaviour. If the agent forgets a constraint mentioned four steps ago, it is because your history trimming dropped it. If it retries the same failing call three times, it is because the failure message did not say why it failed in words the model could act on. Context assembly is not preamble to the interesting work; it is the interesting work.
Write your tool errors as instructions, not as diagnostics. Not `HTTP 404` but `No customer with that ID. Ask the user to confirm the order number.`

### Planning: explicit or emergent
There are two respectable ways to get a plan. Emergent planning lets the model choose one step at a time with no plan document — simple, resilient, and prone to wandering on long tasks. Explicit planning asks for a numbered plan up front, then executes it step by step, re-planning only when a step fails. Explicit planning is easier to audit and much easier to show a user, at the cost of being brittle when reality diverges from step three. For tasks under about five steps, emergent is usually enough; beyond that, an explicit plan pays for itself in traceability.

### Stopping: the part demos never show
| Step cap | 8–15 tool calls | Return partial work with an explanation |
| Spend cap | A fixed cost per run | Stop and log for review |
| Wall clock | 30–120 seconds for interactive use | Hand back with what is known so far |
| Repeat detection | Same call and arguments twice | Force a different branch or stop |
| Human gate | Any irreversible action | Pause and request approval |

### Reading a trace when something goes wrong
A trace is the ordered record of every context, decision, call and result in one run. It is the only debugging tool that matters, and the first thing to build. When an agent misbehaves, the question is never why the model is bad; it is which turn first went wrong and what the model could see at that moment. Nine times out of ten the answer is boring: a tool returned an empty list and said nothing about it, a stale fact stayed in the context, or a permission error was phrased as a generic failure and the model treated it as retryable.

Q: How many steps should an agent take before stopping?
A: For interactive tasks, a cap of eight to twelve tool calls covers almost everything legitimate; a run that needs more is usually stuck. Batch tasks can go higher, but pair a higher cap with a spend cap so a loop cannot be expensive as well as long.

Q: Should the agent plan first or decide step by step?
A: Short tasks do fine deciding one step at a time. Once a task reliably takes more than five steps, an explicit plan makes the run auditable and lets you show progress to a user — re-plan on failure rather than following a stale plan off a cliff.

Q: Why does my agent repeat the same failing call?
A: Almost always because the failure message contains no actionable information. Return short, plain-language errors that state what was wrong and what a sensible next step would be, and add repeat detection so an identical call with identical arguments cannot happen twice in one run.

## AI Agent vs Chatbot: Which One Does Your Problem Actually Need?
https://aiagentdevelopment.info/guides/ai-agent-vs-chatbot
Updated 2026-07-21 · Agent basics

- Chatbots answer; agents change things in systems outside the conversation.
- The difference sets the budget, the testing strategy and who must approve launch.
- Most successful builds are hybrids: retrieval answering plus two or three chosen tools.
- Log what users ask a chatbot to do that it cannot — that is your tool roadmap.

Most teams asking for an agent describe a chatbot, and a good number of teams asking for a chatbot describe an agent. The label matters because the two have almost nothing in common once you get past the text box: different failure modes, different testing, different approvals, different cost curves.

The dividing line is simple. Does the software need to change something outside the conversation? If the answer is no — it explains, summarises, drafts, retrieves — you want a chatbot, possibly with retrieval, and you should be live in weeks. If the answer is yes — it books, refunds, updates, files, sends — you want an agent, and you should plan for months, because the interesting work is in the permissions and the recovery paths, not the replies.

### The honest comparison
| What it produces | Text for a person to read | Changes in a system, plus text |
| Worst realistic failure | A wrong answer someone acts on | A wrong action already taken |
| Testing | Answer quality on a question set | Outcome correctness across whole runs |
| Typical build time | 2–6 weeks | 2–4 months to production |
| Who must approve | Content and support owners | Also security, data and the system owner |
| Ongoing cost driver | Tokens and content upkeep | Integration drift and eval maintenance |

### Signs you actually want a chatbot
- The useful output is an explanation, a summary or a draft a person will review.
- Your knowledge changes more often than your processes do.
- You have no API you would be comfortable letting software write to.
- The value is deflection — fewer easy tickets reaching a human.

### Signs you actually want an agent
- The person reading the answer then goes and does five clicks in another system.
- The task requires looking something up before you know what the next step even is.
- Success is measurable as a completed transaction, not a satisfied reader.
- A human already follows a checklist, and the checklist branches.

### The hybrid that usually wins
The shape that survives contact with real users is rarely pure. It is a chatbot that can call two or three carefully chosen tools, with a human gate in front of anything irreversible. You get the fast path to value from retrieval-based answering, and you add exactly the actions that remove the most clicks — order lookup before returns questions, appointment availability before booking questions. Each added tool is a small, testable increment rather than a leap into full autonomy.
Start by instrumenting the chatbot: log what people ask for that it cannot do. That log is your tool roadmap, ranked by demand rather than by imagination.

### What changes for your team, not just your code
An agent changes who is accountable. A wrong chatbot answer is a content problem, owned by the team that owns the content. A wrong agent action is an operational incident, owned by whoever owns the system it touched — and that team will reasonably want an audit trail, a way to reverse it, and a limit on how much can go wrong per hour. Budget for those conversations at the start of the project. Teams who skip them build a working agent and then spend a quarter failing to get it launched.

Q: Can I upgrade a chatbot into an agent later?
A: Yes, and it is usually the cheapest path. Keep the retrieval layer, the logging and the prompt assets separate from the reply loop, then add tools one at a time with a human approval gate. The parts you would throw away are small; the operational learning you gain from running the chatbot first is not.

Q: Is a chatbot always cheaper to run?
A: Per request, usually yes, because an agent takes several model calls where a chatbot takes one. Per outcome, often no. If an agent completes a task that otherwise costs eight minutes of staff time, the extra tokens are irrelevant next to the labour it displaced.

Q: Which one is riskier for a regulated business?
A: The agent, plainly, because it acts. That does not rule it out — it means the approval gates, audit logging and reversal paths are part of the build rather than a later phase, and it means starting with actions that are reversible by design.

## What Is an AI Agent? A Working Definition for Builders
https://aiagentdevelopment.info/guides/what-is-an-ai-agent
Updated 2026-07-21 · Agent basics

- An AI agent decides, acts through real tools, observes the result and decides again.
- The loop and the tool contracts are the engineering; the model is one component in it.
- Fixed step sequences are workflows — cheaper, more predictable, and often the right answer.
- Autonomy is per-action: draft-only, reversible-only, sandboxed, or unrestricted.

The word agent has been stretched until it covers everything from a prompt with a nice name to a distributed system with its own on-call rota. That is not a vocabulary problem, it is a budgeting problem: teams approve one thing and are handed the other.

Here is the definition we use when scoping work, and it is deliberately narrow. An AI agent is software in which a language model chooses the next step, calls a real tool to take that step, reads what came back, and chooses again — until a goal is met or a limit stops it. If nothing in your system calls a tool, you have a very good text generator. If the sequence of steps is fixed in advance, you have a workflow with a model in one of the boxes. Both are fine things to build. Neither needs an agent budget.

### The loop, not the model, is the product
Every agent is the same three moves repeated: decide, act, observe. The model contributes the decide step. Everything else — what tools exist, how their failures are phrased, what state survives between iterations, when the loop must stop — is ordinary software that you write and own. Teams who believe the model is the product spend their time on prompts and are surprised when the system is unreliable. Teams who treat the loop as the product spend their time on tool contracts and stopping conditions, and get something they can debug on a bad afternoon.
A useful test: if you removed the model and replaced it with a person reading the same information, would the rest of the system still make sense? If not, the surrounding software is too thin.

### What separates an agent from the things it is confused with
| Chatbot | Nobody — it answers what was asked | No | Wrong or invented answer |
| Workflow with an LLM step | The developer, in advance | Yes, in a fixed order | Breaks on inputs outside the flow |
| Agent | The model, at run time | Yes, chosen at run time | Wanders, loops, or acts on bad data |
| Multi-agent system | Several models, plus a coordinator | Yes | All of the above, harder to trace |

### The four parts every real agent has
Strip away the framework names and every production agent we have worked on contains the same four parts.
- A goal that can be checked. Not a vibe: a sentence a reviewer could mark right or wrong.
- A tool surface. The specific functions it may call, with typed arguments and honest error returns.
- A memory or state carrier. What the next iteration is allowed to see from the last one.
- Stopping conditions. A step cap, a spend cap, and a rule for handing control to a human.

### Autonomy is a dial, not a switch
The interesting decision in agent design is not whether to use an agent but how much rope to give it. In practice there are four settings, and most successful projects start further left than the demo suggests: the agent drafts and a human sends; the agent acts on reversible things and asks about irreversible ones; the agent acts freely inside a sandbox with a spend cap; the agent acts freely on production systems. Each step right multiplies both the value and the blast radius. Move right when your evaluation numbers earn it, not when the roadmap says so.

### Where the definition earns its keep
Being strict about the word saves real money in three places. Scoping: a fixed sequence of five API calls with one summarisation step is a workflow, and building it as an agent adds nondeterminism you did not need. Estimation: agents cost more than workflows because the failure surface is larger, and an honest label sets an honest budget. Evaluation: you can only test an agent properly once you accept that the same input may take different paths, which means testing outcomes rather than transcripts.
If a stakeholder asks for an agent, ask what decision they want the software to make on its own. If there is no such decision, you have just saved them three months.

Q: Is a chatbot an AI agent?
A: Not by this definition. A chatbot answers within a conversation; an agent takes actions in systems outside the conversation. A support chatbot that reads your order database, issues a refund and emails the customer is an agent — the actions are what changed its category, not the fact that it talks.

Q: Does an AI agent need to be autonomous to count?
A: It needs to choose its own next step, which is not the same as acting without oversight. An agent that plans five steps, executes four of them and stops for human approval on the fifth is still an agent. Autonomy is a setting you choose per action, based on how reversible that action is.

Q: Do I need a framework to build one?
A: No. The smallest useful agent is a while loop, a list of tool definitions, and a stopping condition — perhaps a hundred lines. Frameworks earn their place when you need durable state, branching control flow or coordination between several agents, not at the start.