Agent Memory & State
The Stateless Core
Here is the single most important fact about LLM-based agents: the language model is stateless. A model call is a pure function — text in, text out. The model retains nothing between calls. It does not "remember" your last message the way a person does. Everything the model is allowed to know on a given turn must be physically present in the text you send it: the context window.
Every "memory" an agent has is something re-supplied into the context window each turn. The illusion of a continuous conversation is created by re-sending the prior turns (or a compressed version of them) on every call. There is no hidden state inside the model carrying your history forward.
The catch is that the context window is finite — a fixed token budget (a few thousand to a few hundred thousand tokens, depending on the model). You cannot just keep appending forever. Managing what goes into that finite window, and pulling in older knowledge when it becomes relevant, is the entire discipline of agent memory. If you are new to how text becomes vectors, see Vector Embeddings and Agent Architectures.
A Taxonomy of Memory
Borrowing loosely from cognitive science, agent memory splits into two tiers by where it lives and how long it lasts.
Short-term / working memory
The conversation currently inside the context window. Fast, immediately available to the model, but bounded by the token budget and gone once the session ends (or once it is evicted to make room). This is the agent's "RAM".
Long-term memory
An external store that persists across turns and across sessions — typically a vector database. It is effectively unbounded, but it is not in context: the model cannot see it until relevant pieces are retrieved and injected into the window. This is the agent's "disk".
Three flavors of long-term memory
Long-term memory is usually subdivided by what kind of thing is stored:
| Type | Stores | Example |
|---|---|---|
| Episodic | Past interactions / events — what happened, and when | "Last Tuesday the user asked to book a flight to Paris." |
| Semantic | Facts / knowledge, decontextualized from any one event | "The user is vegan and allergic to peanuts." |
| Procedural | Skills / instructions — how to do things | "When booking, always confirm the date before paying." (often the system prompt / tools) |
Mnemonic: episodic = "I remember when it happened," semantic = "I know that it is true," procedural = "I know how to do it."
The Core Technique: Retrieve Into Context
Because the model can only act on what is in the window, long-term memory is useless unless you can pull the right pieces back in at the right moment. The dominant technique is RAG over memory:
- Store as embeddings. Each memory (a fact, a past turn, a summary) is converted into a vector with an embedding model and saved in a vector store.
- Embed the new query. When a new turn arrives, embed it into the same vector space.
- Retrieve by similarity. Find the stored memories whose vectors are closest to the query (typically by cosine similarity), and take the top-k.
- Inject into context. Paste those retrieved memories into the window alongside the new turn, then call the model.
Similarity is usually cos(θ) = (a·b) / (‖a‖ ‖b‖), where a·b is the dot product and ‖a‖ is the vector's length (its L2 norm). It ranges from −1 (opposite) to +1 (identical direction); higher means "more relevant." The agent retrieves the top-k highest-scoring memories. This is exactly retrieval-augmented generation, but the corpus being searched is the agent's own memory.
The slogan to keep: long-term memory = retrieve-into-context. The store can be enormous, but only a small, relevant slice ever enters the finite window per turn.
Interactive: Context Window & Memory Retrieval
Step through a multi-turn conversation. Each turn is appended to a fixed-size context window (the token-budget bar). When the window overflows, the strategy you select keeps it within budget — by throwing old turns away, compressing them into a summary, or offloading them to a long-term vector store and retrieving only the relevant items back when a query needs them. Watch the final question ("what is my dog called?") to see which strategy still has the answer.
The bar is the finite context window. Every turn must fit inside it — that is the fundamental bottleneck.
In context (short-term / working memory)
Empty. Click "Add next turn".
Long-term memory store (vectors)
Empty. Only the "Offload + retrieve" strategy uses an external store.
Vector dims: [travel, food, work, personal]. Retrieval ranks by cosine similarity to the new query.
Context-Management Strategies
Keeping a long conversation inside a finite window is a constant balancing act between cost (fewer tokens) and fidelity (keeping the information that matters). The main levers:
| Strategy | How it works | Trade-off |
|---|---|---|
| Truncation / sliding window | Keep only the most recent N turns; drop the oldest. | Cheap and simple, but early facts are lost permanently. |
| Summarization / compaction | Replace a block of old turns with a short LLM-written summary. | Keeps the gist in few tokens; specific details can be lost. |
| Offload + retrieve (RAG over memory) | Move old turns to a vector store; retrieve relevant ones on demand. | Nothing is lost; adds retrieval cost and can miss-rank. |
| Importance-based retention | Score memories (recency, frequency, salience) and keep/evict by score. | Keeps high-value items; scoring heuristic can be wrong. |
These compose. A common production setup: a sliding window of recent turns kept verbatim, a rolling summary of everything older, and a vector store queried each turn to pull back any specific older detail the summary dropped.
Scratchpads & Task State
Conversation history is not the only thing an agent must track. For multi-step tasks, agents maintain explicit state — a structured scratchpad that records the plan, intermediate results, what has been done, and what remains. Because the model is stateless, this scratchpad lives outside the model (in the orchestration code or a file) and is re-injected into the context each turn so the agent can pick up where it left off.
Scratchpad contents
- The current goal and sub-goals
- A running plan / to-do checklist
- Intermediate results and tool outputs
- Open questions or blockers
Why keep it external
- It survives across stateless model calls
- It can be compacted independently of chat history
- It can be inspected, edited, and resumed
- It keeps reasoning structured, not buried in prose
Common Pitfalls
Assuming the model "remembers." It does not. If a fact is not in the window (or retrievable into it), the model genuinely cannot use it — no matter how many times you said it earlier.
Stuffing the whole window. A bigger context is not free: it costs more, can be slower, and models often attend poorly to information buried in the middle of a very long context ("lost in the middle"). Curating what you retrieve usually beats retrieving more.
Over-aggressive summarization. Compress too hard and you lose the one specific detail (a name, a number, an allergy) you will need later. Pair summaries with a vector store so specifics remain recoverable.
Key Takeaways
- LLMs are stateless per call: all "memory" must be supplied in the context window each turn, and that window is finite — the fundamental bottleneck.
- Short-term / working memory = the conversation inside the context window (fast, bounded, transient). Long-term memory = an external store (usually a vector DB) that persists across sessions but is invisible until retrieved.
- Long-term memory comes in three flavors: episodic (past events), semantic(facts), and procedural (skills / instructions).
- The core technique is RAG over memory: store memories as embeddings, embed the new query, retrieve the most similar items by cosine similarity, and inject them into context. Long-term memory = retrieve-into-context.
- Manage the finite window with truncation / sliding windows, summarization / compaction, offload-and-retrieve, and importance-based retention — often combined.
- For multi-step tasks, keep an external scratchpad / state object and re-inject it each turn, since the model carries nothing forward on its own.