Agent memory needs three layers: session state for now, vector stores for recall, and explicit summaries for durable intent in production. Full breakdown.
Why One Memory Layer Is Never Enough
An agent that keeps everything in a single store eventually fails in one of two directions. Stuff every message into the prompt and you run out of context, pay for tokens you don't need, and bury the instructions that actually matter under transcript noise. Push everything into a vector database and you lose the precise, ordered state the agent needs to finish the task it is working on right now. The fix is to stop treating memory as one thing. Separate it by how long the information needs to live and how it will be retrieved.
Three layers cover the practical cases: session state for what is happening now, vector stores for recall of things said or seen before, and explicit summaries for durable intent that must survive across sessions. Each has a different write pattern, a different read pattern, and a different failure mode.
Session State: The Working Set
Session state is the agent's short-term memory — the current goal, the last few tool results, variables it just computed, and the running plan. It is ordered, it is exact, and it belongs in the context window because the model reasons over it directly on every step. Treat it as the working set, not an archive: keep it small enough to stay fast and cheap, and drop or compress entries as they stop being relevant to the active task.
The discipline here is eviction. When session state grows past what the task needs, decide deliberately what to keep in the prompt versus what to hand off to one of the other layers, rather than letting the transcript accumulate until it overflows.
Vector Stores: Recall Without Rehydrating Everything
Vector stores answer a different question: "have I seen something like this before?" You embed documents, past conversations, or tool outputs, and retrieve the closest matches on demand instead of holding them all in context. This is how an agent recalls a fact from three sessions ago without replaying three sessions of history.
The tradeoff is that retrieval is approximate. Similarity search returns what looks related, not what is guaranteed relevant, so quality depends on what you choose to embed and how you chunk it. A few practices that hold regardless of the specific stack:
- Store retrievable, self-contained chunks — a fragment that only makes sense with its neighbors will mislead when pulled out alone.
- Attach metadata (source, timestamp, task) so you can filter before ranking by similarity.
- Feed retrieved results back into session state as context, not as commands the agent must obey.
Explicit Summaries: Durable Intent
The layer teams most often skip is the explicit summary: a deliberately written record of what the agent is trying to accomplish, the decisions already made, and the constraints it must respect. Unlike vector recall, this is not fuzzy — it is a compact, authoritative statement of intent that you carry forward on purpose. It is what lets an agent resume a long-running goal after the session state has been evicted and the raw transcript is gone.
Write these summaries at natural boundaries: when a task phase completes, when context is about to be compressed, or when a session ends. Keep them terse and unambiguous, because they will be re-read as ground truth. In production, the combination matters most — session state keeps the agent coherent now, vectors let it recall the past, and summaries keep it pointed at the right goal across the gaps between.