Memory in AI Agents: What to Keep, Compress and Throw Away
There is no memory in a language model call. Each turn is a fresh request, and the only thing the model knows is what you assembled into its context this time. Everything people describe as an agent forgetting, or getting worse over a long run, is a decision your code made about what to carry.
Once you accept that, memory design becomes an ordinary engineering problem with a familiar shape: what is always relevant, what is recently relevant, what can be summarised, and what should be fetched fresh instead of stored at all.
The four tiers#
| Tier | Content | Trimmed? | Typical size |
|---|---|---|---|
| Pinned | Goal, constraints, user identity, policy | Never | 200–500 tokens |
| Recent | Last few turns verbatim, with tool results | Rolling window | 2–5 turns |
| Compressed | Older turns as short factual notes | Rewritten as it grows | Under 500 tokens |
| Retrieved | Documents and records fetched for this step | Discarded after use | Per step |
Compress facts, not prose#
The usual mistake is to summarise old turns as narrative — the user asked about their order and the agent looked it up. That reads well and helps nothing. Compress into the facts a later step might need: order 4471, status shipped, customer requested refund, refund policy allows 30 days, no refund issued yet. Structured, checkable, and a tenth of the size. If a later step is going to make a decision, the compression must preserve the inputs to that decision or it has failed at its only job.
Compress on a threshold, not every turn. Re-summarising a summary repeatedly is how details quietly disappear.
Retrieval is not memory#
Documents pulled from a knowledge base belong to the step that needed them. Keeping them in the running context after that step is finished is the fastest way to a bloated, expensive, distracted run. Fetch, use, cite, discard — and if a later step needs the same fact, fetch it again. Retrieval is cheap; a context full of stale documents is not.
Memory that persists between sessions#
Long-lived agents accumulate genuinely useful facts about a user or account: preferences, prior decisions, constraints that will not change. Store these deliberately, in a small structured record with an explicit write step, rather than by letting conversation history pile up. Three rules keep it healthy: write only facts a future run would act on, always record where a fact came from, and give every fact an expiry or a review date. Persistent memory without provenance and expiry becomes a slowly rotting source of confident errors.
- Write on purpose — a tool call, not a side effect of chatting.
- Store the source and the date alongside every fact.
- Cap the record size and expire what has not been used.
- Let the user see and correct what is stored about them.
Symptoms and their causes#
| Symptom | Usual cause |
|---|---|
| Forgets a constraint from early in the run | Constraint was not pinned; it was trimmed with the history |
| Quality degrades after several steps | Context is diluted with stale tool output |
| Repeats a completed step | Result was summarised away without an outcome marker |
| Cost climbs with run length | Retrieved documents are accumulating instead of being discarded |
| Confidently states something outdated | Persistent memory has no expiry or provenance |
Frequently asked questions
How much history should I keep verbatim?
Three to five turns covers most reasoning without dominating the context. Keep tool results attached to their calls, and compress anything older into structured facts rather than dropping it silently.
Should I use a vector database for agent memory?
For retrieving documents, often yes. For the running state of a single run, no — that is a small structured object in your own store. Conflating the two produces both a fuzzy state machine and an unfocused search index.
How do I stop persistent memory from going stale?
Give every stored fact a source, a date and an expiry, and make the agent prefer freshly retrieved data when both exist. Expose the stored record to the user so wrong facts can be corrected rather than silently repeated.
ai agent memorycontext managementconversation summarisationpersistent agent memoryllm context window
