Hiring AI Agent Developers: What to Look For and How to Test
The job title is new, the skill set is not. The people who build agents that survive production are ordinary strong engineers who have learned to work with a component that is fast, capable and occasionally confidently wrong.
That reframing makes hiring much easier. You are not looking for a prompt specialist. You are looking for someone who instinctively asks what happens when the tool returns nothing, and who has an opinion about how you would know the change made things better.
What matters, in order#
- API and integration engineering: most of the work is talking to your systems properly.
- Testing instinct: they ask about evaluation before they ask about the model.
- Failure-mode thinking: empty results, permissions, timeouts, partial success.
- Security awareness: least privilege, injection, audit trails, approval gates.
- Cost awareness: they can explain where tokens go without looking it up.
- Model familiarity: useful, learnable in weeks by a strong engineer.
- Framework knowledge: the least important item, and the most advertised.
A ninety-minute screening exercise that works#
Give a candidate a short brief: an agent that answers order questions and can issue refunds under fifty pounds. Ask for the tool list with argument schemas, the stopping conditions, ten evaluation cases, and what sits behind a human gate. You are not looking for code. You are looking for whether they define `refund_order(order_id)` rather than `update_order(order_id, fields)`, whether they include a should-refuse case and an empty-result case, and whether the refund gate appears without being prompted.
Strong candidates ask clarifying questions about permissions and edge cases in the first five minutes. It is the most reliable signal in the whole process.
Questions that separate experience from enthusiasm#
| Question | Weak answer | Strong answer |
|---|---|---|
| How do you know a change helped? | We test it manually | A fixed evaluation set, run before and after |
| What do you do when a tool returns nothing? | Retry | Return an explicit empty result the agent can act on |
| How do you stop injection? | Instruct the model to ignore it | Least privilege, content isolation, approval gates |
| Why was your last agent slow? | The model was slow | Six serial calls; parallelised two, cut context |
| How do you pick a model? | The best one | Per step, measured on our own cases |
Agency, contractor or in-house#
An agency suits a first build with a deadline: you buy a team that has already made the standard mistakes, and you should require handover of the evaluation set and tool schemas as a deliverable. A contractor suits extending a system your team will keep. In-house is right when the agent becomes part of the product — at that point someone must own it permanently, and that person needs the context that only comes from building it. The common failure is an agency build with no handover, leaving a system nobody internally can change.
Red flags on both sides#
- A proposal with no evaluation line, or where evaluation means the developers trying it.
- Confidence about accuracy before seeing your data.
- A framework recommendation before the tool list is written.
- No question about permissions or who the end user is.
- Unwillingness to hand over prompts, schemas and evaluation cases at the end.
Frequently asked questions
Do I need a machine learning engineer?
Usually not. Agent work is systems engineering against a model API. Bring in ML expertise when you are fine-tuning, training a classifier, or doing serious retrieval optimisation — not to build the agent itself.
How big should the team be?
Two engineers and a part-time domain expert covers most first builds. The domain expert is not optional: they supply the evaluation cases and decide what a correct outcome looks like, and no amount of engineering substitutes for that.
What should an agency hand over?
Repository, prompts, tool schemas, the evaluation set with results, traces from the last month, a monitoring dashboard, and a written note of known failure modes. If any of these are missing, you have bought a system you cannot safely change.
hire ai agent developersai agency selectionllm engineer interviewagent team structureai development handover
