Agent Memory

An agent's only native memory is its context window, and that window is finite. A long-running task produces far more tool results, reasoning, and history than will fit: a coding agent can burn through 200,000 tokens in under an hour of file reads and test runs. Stuffing everything in degrades quality anyway, because models attend less reliably as context grows. Agent memory is the set of techniques for deciding what stays in the window, what moves outside it, and how the right pieces get pulled back in. It is less a feature than a budgeting discipline for a scarce resource.

Three tiers, three lifespans#

It helps to sort memory by how long it needs to live. Working context is whatever is in the window right now: the system prompt, recent messages, the last few tool results. It is fast and complete but evaporates the moment the session ends or the window fills. A scratchpad is short-term state the agent writes outside the window during a single task: a NOTES.md file, a to-do list, a progress log updated via tool calling. Claude Code does exactly this, and Anthropic's Claude-plays-Pokemon experiment showed an agent keeping accurate tallies across thousands of game steps by maintaining notes no single window could hold. Episodic and long-term memory persist across sessions: summaries of past runs, learned facts, user preferences, self-critiques. Reflexion-style agents write post-mortems into an episodic buffer so the next attempt starts wiser instead of repeating the same failure.

The MemGPT paper (Packer et al., 2023) made this hierarchy explicit by borrowing the operating system analogy: treat the context window like RAM and external storage like disk, and let the agent itself page data between them through tool calls. Letta, the company that grew out of MemGPT, ships it as three named tiers: core memory pinned in context like RAM, recall memory holding searchable conversation history, and archival memory in an external vector store the agent queries explicitly.

Compaction: making room without amnesia#

When the window fills mid-task, the first lever is compaction: summarize the conversation so far and restart with the distilled version. Claude Code does this automatically as it approaches the limit, and the craft is in what the summary keeps. Anthropic's guidance is to preserve architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs, which are usually the bulk of the tokens. The Claude API now offers this server-side, and a lighter-touch variant, clearing old tool results while keeping the messages around them, recovers most of the space at far less risk.

Compaction is lossy by construction. A summary that drops one constraint ("the client requires Python 3.9") can send the next hundred steps in the wrong direction. That is why compaction pairs naturally with the scratchpad: notes written to durable files survive compaction boundaries verbatim, so the summary only has to carry narrative, not facts. Anthropic's memory tool documentation makes the assumption brutal and explicit, instructing the agent to work as if "your context window might be reset at any moment."

Persist or recompute#

Every piece of state forces a choice: save it or rebuild it on demand. The trade is the same as caching anywhere. Persisting saves work: re-deriving a codebase map might cost 50 file reads, and a debugging conclusion that took twenty minutes of tool calls is criminal to throw away. But persisted state goes stale silently, and an agent that trusts last week's notes about a renamed module will confidently edit the wrong file. Stale memory does not fail loudly the way a missing file does; it poisons downstream reasoning.

A workable rule: persist what is expensive to rediscover and slow to change (design decisions, user preferences, hard-won diagnoses, project conventions), and recompute what is cheap or volatile (current file contents, test results, anything a single tool call can refresh). When in doubt, store the pointer rather than the value: "auth logic lives in src/auth/" ages far better than a cached copy of the code itself.

Memory is a retrieval problem#

Once memory lives outside the window, getting it back in is search, the same problem RAG solves, just pointed at the agent's own past instead of a document corpus. Small memories can be loaded wholesale: Claude's memory tool simply has the agent list a /memories directory and read the relevant files before starting work, a pattern Anthropic calls just-in-time retrieval. Past a few dozen notes, you index them like any corpus, embed and search, and Letta exposes exactly that as an archival search tool the agent calls when it decides it needs history.

This framing settles most design arguments. Bigger context windows do not eliminate memory, because cost and attention quality still degrade with length; they just raise the threshold where retrieval beats inlining. And "remember everything" is not the goal: an agent that reloads its full history every session is paying for tokens that actively distract the model. The goal is surfacing the right 2,000 tokens of past experience at the step where they matter.

Practical takeaways#

Start with the boring tier: a scratchpad file plus automatic compaction covers most single-task agents, and both ship out of the box in modern harnesses. Add cross-session memory only when the workflow genuinely recurs, and make it file-based and inspectable first; you can graduate to an embedded store when volume demands it. Decide persist-versus-recompute deliberately, defaulting to recompute for anything cheap. Write summaries that privilege decisions and open questions over chronology. And treat memory quality as a retrieval metric: if the agent cannot pull the right note back at the right moment, it does not matter how faithfully the note was written.