RAG for Agents: Grounding Answers Without Drowning in Context
Retrieval-augmented generation is usually introduced as a pipeline: embed the question, fetch the top chunks, paste them in, generate. That works for a question-answering box. Inside an agent it is the wrong shape, because the agent does not yet know what it needs until it has taken a step.
The version that works treats retrieval as a tool the agent calls when it decides it needs evidence — sometimes twice with different queries, sometimes not at all. That one change removes a great deal of irrelevant context and makes the whole run cheaper and sharper.
Retrieval as a tool, not a preamble#
Expose search as a normal tool with a query argument and a small, structured return: a handful of passages, each with an identifier and a source. The agent decides when to call it, can refine its query after seeing what came back, and can call a different tool instead when the answer is structured data rather than prose. The pipeline version cannot do any of that, and it pays the retrieval cost on every request whether or not the request needed evidence.
Log the queries the agent writes. They are the most honest description you will ever get of what your users are actually asking.
Chunking decisions that matter more than the embedding model#
- Split on structure — headings, sections, list items — not on a fixed character count.
- Keep each chunk self-contained: a chunk that begins with `It also requires` is useless out of context.
- Attach the document title and section heading to every chunk, both for the model and for citation.
- Store an identifier and a URL with each chunk so an answer can point at its source.
- Prefer fewer, larger, meaningful chunks over many small ones; overlap is a patch for bad boundaries, not a strategy.
Hybrid retrieval beats pure vectors on real corpora#
| Query type | Vector search | Keyword search | Best |
|---|---|---|---|
| Conceptual question | Strong | Weak | Vector |
| Exact product code or error string | Weak | Strong | Keyword |
| Rare proper noun | Mixed | Strong | Keyword |
| Paraphrased policy question | Strong | Weak | Vector |
| Most real traffic | Mixed | Mixed | Both, merged and re-ranked |
Make the agent cite, and make it able to fail#
Two requirements do most of the work for trustworthiness. First, every claim drawn from retrieval carries the identifier of the passage it came from, and your interface renders that as a link — this makes unsupported statements visible instead of plausible. Second, the search tool must be able to return nothing, and the agent must be taught that answering `I could not find that in our documentation` is a correct outcome. An agent that cannot fail at retrieval will invent, because inventing is the only remaining option you left it.
Keeping the index honest#
Retrieval quality decays quietly. Documents change, sections are deleted, and the index keeps serving what it last saw. Re-index on a schedule, delete chunks whose source document is gone rather than letting them linger, and keep a small evaluation set of queries with known correct passages so you can measure recall after each change. The failure mode without this is the worst kind: an agent confidently citing a policy that was withdrawn in March.
Frequently asked questions
Should the agent always retrieve before answering?
No. Retrieval on every request wastes latency and fills context on questions that do not need evidence. Let the agent call the search tool when it judges evidence is needed, and measure how often it should have and did not.
How many passages should I return?
Three to six well-chosen passages beat twenty. More retrieved text dilutes attention, raises cost and increases the chance the agent grounds its answer in a passage that merely looked relevant.
What if retrieval returns nothing useful?
That must be a supported outcome. Return an explicit empty result, and instruct the agent to say it could not find the information and offer the next step — asking a clarifying question or handing over to a person.
rag for agentsretrieval augmented generationhybrid searchchunking strategygrounded answers
