Memory: Short-Term, Long-Term and Notes

The agent that forgets everything by default

In part 2 we built the agent loop: observe, think, act, repeat. Each time the loop runs, the agent sends the large language model a prompt built from whatever text is currently available to it, and the model produces the next thought or action based only on that text. This is worth sitting with for a moment, because it explains almost every memory problem an agent has. The model itself does not remember anything between calls. It has no notebook, no database, no recollection of the user it spoke to yesterday. Every single time you call it, it starts from zero and reconstructs its understanding purely from the text you hand it.

This means that anything an agent appears to remember, it remembers because someone engineered a way to put that information back in front of the model at the right time. There are three broad ways to do this, and they solve different problems: short-term memory, which is the running conversation and scratch material inside the current context window; long-term memory, which is information stored outside the model and retrieved on demand, often across entirely separate sessions; and notes, a more deliberate, agent-authored form of memory that sits between the two and is especially useful for long or multi-step tasks like the ones we decomposed in part 5. This article works through all three, with concrete numbers, so that by the end you can reason about which kind of memory a given problem actually needs rather than reaching for a vector database because it sounds sophisticated.

Short-term memory: the context window

The context window is the amount of text, measured in tokens, that a model can consider at once in a single call. A token is roughly three to four characters of English text, or very roughly three quarters of a word, though the exact split depends on the tokenizer the model uses. A model with a context window of 128,000 tokens can consider something in the rough range of 90,000 to 100,000 words in one call, including both the input you send and the output it produces. This number varies by model and provider, and context window sizes have grown substantially over the last few years, so treat any specific figure here as illustrative rather than a claim about a particular model you might be using.

In an agent loop, the context window is where short-term memory lives. Every message the agent has seen so far in the current run, every tool call it made, every tool result that came back, all of it typically gets appended to a running list of messages and sent back to the model on the next turn, so the model appears to remember what happened three steps ago. It does not remember it in the human sense; it is simply being shown the transcript again, in full, every single time.

Watching the context grow

It helps to see this accumulate with actual numbers. Suppose an agent is working through a task that involves calling a web search tool, as introduced in part 3, and reading back the results. A short user question might run 20 tokens. A tool call the model emits might be 15 tokens. A tool result, if it is a paragraph of search snippets, might run 400 to 800 tokens. If the agent calls three tools in sequence before answering, the running transcript might look like this by the end: system prompt 300 tokens, user question 20 tokens, first tool call 15 tokens, first tool result 600 tokens, second tool call 15 tokens, second tool result 550 tokens, third tool call 15 tokens, third tool result 700 tokens, and finally the model's answer 150 tokens. Add that up and you are already past 2,300 tokens for what looks, from the outside, like a short three-step task. Repeat this pattern for ten loop iterations instead of three, or let a tool return a full web page instead of a snippet, and you can reach tens of thousands of tokens before the agent has done anything especially complicated.

# A rough, illustrative way to track context growth in an agent loop.
# This uses a word-count approximation for tokens (roughly tokens ~= 1.3x words
# for English text with common tokenizers); use a real tokenizer such as
# tiktoken for an accurate count in a production setting.

def approx_tokens(text: str) -> int:
    words = len(text.split())
    return int(words * 1.3)

transcript = []

def add_message(role: str, content: str):
    transcript.append({"role": role, "content": content})
    total = sum(approx_tokens(m["content"]) for m in transcript)
    print(f"after {role} message: ~{total} tokens in context")

add_message("system", "You are a research assistant. Use tools when needed.")
add_message("user", "Find out when the Eiffel Tower was completed and by whom.")
add_message("assistant", "call tool: web_search(query='Eiffel Tower completion date architect')")
add_message("tool", "The Eiffel Tower was designed by Gustave Eiffel's engineering company " * 20)
add_message("assistant", "The Eiffel Tower was completed in 1889 and designed by Gustave Eiffel's company.")

Running this prints a growing token count after each message. The point of the exercise is not the exact numbers, which depend on the real tool output and the real tokenizer, but the shape of the curve: it only goes up. Nothing in the basic loop from part 2 removes anything from the transcript. If you do not actively manage this, a long-running agent will eventually either exceed the context window and fail outright, or stay within it but become slow and expensive because every single call now re-processes thousands of tokens of history the model does not actually need anymore.

Why a bigger context window does not fully solve this

It is tempting to think that context window growth makes this whole problem go away. If models can now handle context windows in the hundreds of thousands of tokens, why not just let the transcript grow and let the model sort it out? Three practical reasons argue against this. First, cost: most providers charge per token processed, and a model that re-reads 50,000 tokens of history on every single loop iteration before producing one short answer is paying that cost repeatedly, not once. Second, latency: more input tokens generally means more time before the first output token appears, which matters if a human is waiting on the other end. Third, and more subtly, some research has reported that models do not use very long contexts uniformly well; information placed in the middle of a very long prompt can be attended to less reliably than information near the start or the end, an effect sometimes described informally as information getting "lost in the middle." The exact severity of this effect varies by model and is an active area of study rather than a settled number, so treat it as a reason to be careful rather than a precise rule, but the practical implication is consistent: a bigger window raises the ceiling, it does not mean you should fill it carelessly.

Managing short-term memory: truncation, windows and summarization

Because the transcript only grows, an agent that runs for more than a handful of loop iterations needs an explicit strategy for keeping the context under control. There is no single correct strategy; the right one depends on the task. Three common approaches, often combined, are worth knowing.

• Fixed truncation: keep the system prompt and the most recent N messages, and drop everything older. Simple to implement, but it can discard information the agent genuinely still needs, for example a fact established ten turns ago that becomes relevant again later.
• Sliding window with anchors: keep the system prompt, the original user request, and the most recent K turns, dropping only the middle. This preserves the goal of the task even as the recent detail changes, which matters for long tasks where the agent must not forget what it was originally asked to do.
• Rolling summarization: periodically ask the model (or a separate, cheaper call) to compress older turns into a short summary, then replace those turns with the summary in the transcript. This keeps the gist while shrinking the token count substantially.

A worked example of rolling summarization makes the trade-off concrete. Suppose the agent has just finished a tool call whose result was a 600-token block of search snippets about three possible suppliers for a product. Once the agent has extracted what it needs, say the three supplier names and their quoted prices, keeping all 600 tokens around for the rest of the run is wasteful. A summarization step could reduce that to something like: "Search found three suppliers: Acme (12 dollars per unit, 2 week lead time), Brightline (9 dollars per unit, 4 week lead time), Carno (15 dollars per unit, 1 week lead time)." That is roughly 35 tokens carrying forward the only facts that matter, a reduction of more than fifteen times from the original tool output. The cost of this approach is an extra model call to do the summarizing, and a small risk that the summary drops something that turns out to matter later, which is why summarization is best applied to information the task has clearly already used, not to the most recent, still-relevant turns.

None of these strategies are about erasing memory entirely; they are about deciding, on purpose, what stays inside the window the model actually sees. That decision, of what to keep, drop, or compress, is itself the subject of context engineering, which is where part 7 goes next. For this article, the important idea is simpler: short-term memory is not infinite, it is not free, and an agent that works for more than a few steps needs a plan for it.

Long-term memory: remembering across sessions

Short-term memory, however well managed, disappears the moment the run ends. Close the conversation, start a new one tomorrow, and none of yesterday's context window exists anymore. That is fine for a one-off task, but it falls apart for anything an agent is supposed to handle repeatedly: a coding assistant that should remember your team's style conventions, a customer support agent that should remember a customer already explained their problem last week, or a personal assistant that should remember you prefer morning meetings. These require long-term memory: information stored somewhere outside the model and the current context window, and deliberately retrieved and inserted back into the prompt when it becomes relevant.

The mechanics are simple to state even though building them well is not trivial. Something, at some point, decides that a piece of information is worth keeping (a fact, a preference, an outcome). That piece of text gets stored in a persistent store, often alongside a numerical representation of its meaning. Later, in a different session, before or during the agent loop, the system searches that store for entries related to the current situation and inserts the relevant ones into the prompt as additional context, exactly as if they were a tool result. The agent model itself does nothing special here; from its point of view, retrieved memories just look like more text in the prompt, the same way a tool call result does.

Three kinds of long-term memory

It helps to separate long-term memory into three categories, borrowed loosely from how memory is sometimes categorised in cognitive psychology, because each one is stored and used a little differently in practice.

• Episodic memory: records of specific past events. Example: "On March 3rd, the user asked to cancel order #4821 and the agent confirmed the cancellation." This is useful for continuity, so a support agent does not ask a customer to repeat something they already said.
• Semantic memory: general facts, independent of when they were learned. Example: "The user prefers metric units" or "The company's refund policy allows returns within 30 days." This is useful for applying consistent facts and preferences across many separate interactions.
• Procedural memory: knowledge of how to do something, often learned from past successes or failures rather than stated outright. Example: "When the invoice API returns a 429 error, wait 5 seconds and retry rather than treating it as a hard failure," learned after the agent previously handled that situation correctly. This is closer to a reusable strategy than a fact.

These categories are a useful way to think about what you are storing and why, not a mandate to build three separate databases. A small system might lump all of these into one store with a tag describing the type. A larger system might keep them genuinely separate, because an episodic record of "what happened" and a semantic fact about "what is true" are retrieved differently and go stale differently; an episodic memory is valid forever as a record of what occurred, while a semantic memory like a stated preference can become outdated and needs a way to be corrected or replaced.

How retrieval actually works: embeddings and similarity

The retrieval step, finding which stored memories are relevant to the current situation, is usually done with embeddings rather than exact keyword matching, because the agent needs to find memories that are related in meaning even if the wording is completely different. An embedding is a list of numbers produced by a model, where texts with similar meaning end up with similar lists of numbers. Real embedding models produce vectors with hundreds or thousands of numbers; to see the idea clearly, it helps to shrink that down to three numbers and work through it by hand.

Imagine, purely as an illustration, that a simplified embedding model represents short texts as 3-dimensional vectors, where the three dimensions loosely correspond to "about travel," "about food," and "about technology" (real embeddings do not have clean, labelled dimensions like this; this is a teaching simplification, not how embeddings actually work internally). Suppose three memories have been stored with these toy vectors: "User enjoys hiking in the mountains" as [0.9, 0.1, 0.0], "User prefers vegetarian restaurants" as [0.2, 0.9, 0.0], and "User works as a backend developer" as [0.0, 0.0, 0.95]. Now the current user message is "Can you recommend a good trail for this weekend?", which the same embedding model turns into, say, [0.85, 0.05, 0.1]. To find the most relevant memory, the system compares this new vector against each stored vector using cosine similarity, a measure of how closely two vectors point in the same direction, ranging from -1 for opposite directions to 1 for identical direction.

import numpy as np

def cosine_similarity(a, b):
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

query = [0.85, 0.05, 0.1]

memories = {
    "User enjoys hiking in the mountains": [0.9, 0.1, 0.0],
    "User prefers vegetarian restaurants": [0.2, 0.9, 0.0],
    "User works as a backend developer": [0.0, 0.0, 0.95],
}

for text, vector in memories.items():
    score = cosine_similarity(query, vector)
    print(f"{score:.3f}  {text}")

Running this computes a similarity score for each memory against the query. The hiking memory, whose vector points in almost the same direction as the query, comes out with the highest score, close to the top of the -1 to 1 range; the vegetarian restaurant memory scores noticeably lower because it points in a largely different direction; the backend developer memory scores lowest, close to zero, because it has almost nothing in common with the query vector. A real memory system does exactly this at larger scale: it stores vectors from an actual embedding model, with hundreds of dimensions instead of three, and uses a vector database or a library built for approximate nearest-neighbour search to find the closest handful of stored vectors out of potentially millions, quickly enough to use in an interactive agent loop. The underlying operation, computing how aligned two vectors are and ranking by the result, is the same idea you just worked through by hand; the scale and the dimensionality are the only things that change.

What belongs in long-term memory, and what does not

Storing everything is tempting because storage is cheap, but it creates two real problems. The first is retrieval quality: a memory store cluttered with low-value entries, like the full text of every tool call an agent ever made, makes it harder for similarity search to surface the handful of entries that actually matter, because they get diluted by noise with superficially similar wording. The second is correctness over time: memories can become stale or contradictory, and a store with no way to update or retire an old entry will eventually return outdated information as if it were current, for example recommending a restaurant the user mentioned disliking three months after they said so, because the earlier positive mention was never corrected.

A reasonable rule of thumb is to store things that are durable and reusable across sessions, such as explicit user preferences, corrections the user made to the agent's behaviour, stable facts about the domain, and significant outcomes worth recalling later, and to avoid storing routine, one-off operational detail, like the full text of every intermediate tool result, which belongs in short-term memory or nowhere at all once the task that needed it is done. When a new memory contradicts an old one, for example the user previously said they live in Berlin and now says they have moved to Lisbon, the system needs a deliberate way to handle that, either by updating the old entry or by storing the new one with a timestamp and preferring the most recent entry at retrieval time. Getting this wrong quietly, by never updating anything, is one of the more common failure modes in systems that add long-term memory without thinking through its lifecycle.

Notes: the agent's own working memory

Short-term memory and long-term memory solve two different problems: what the model sees right now, and what persists across sessions. There is a third kind of memory that solves a problem neither of the other two handles well: keeping an agent oriented and non-repetitive during a single long or complex task. This is usually called a scratchpad or notes, and it is closely connected to the planning work from part 5.

Think back to task decomposition: an agent given a multi-step goal often benefits from writing out a plan before acting, something like "1) search for candidate suppliers, 2) compare prices and lead times, 3) check each supplier's return policy, 4) recommend the best option with reasoning." That plan is itself a form of memory. It is not part of the model's weights, it is not a fact retrieved from a long-term store, and it will not persist past this task unless something saves it. It exists purely as text that the agent wrote for itself, to be read back later in the same task.

Notes extend this idea beyond the initial plan into ongoing bookkeeping during execution. As the agent works through the plan, it can update its notes: "Step 1 done: found 4 candidate suppliers (Acme, Brightline, Carno, Delmax). Step 2 in progress: Acme and Brightline compared so far, Carno and Delmax still to check." The next time the loop runs, instead of re-reading the entire transcript of every search result that led to this state, the agent (or the system around it) can read this short note and know exactly where it left off.

Why not just rely on the context window for this

If the full transcript is already in the context window, why bother writing a separate note that essentially restates it? Three practical reasons. First, compression: a two-sentence note capturing the state of a four-step plan is far cheaper, in tokens, than the full transcript of the tool calls and results that produced that state, which matters directly for the cost and latency issues discussed earlier in this article. Second, resilience to trimming: if the agent's context management strategy truncates or summarizes older turns, as discussed above, a plan buried in message fifteen can get lost or garbled in that process, while an explicit, actively maintained note is far less likely to be dropped because it is deliberately kept current, often by being rewritten each time rather than simply appended to. Third, and perhaps most practically, notes allow a task to be paused and resumed, even across separate sessions, in a way raw conversation history does not: if the note is saved to a file or database, a run that got interrupted after step 2 of 4 can start a new session, read the note, and continue from step 3 instead of restarting the whole plan or losing track entirely.

In practice, notes are often implemented as nothing more exotic than a text file, or a small set of key-value entries, that the agent reads and writes to using ordinary tools, in exactly the sense covered in part 3. A write_notes tool might take a block of text and overwrite or append to a stored scratchpad; a read_notes tool returns the current contents. From the model's perspective, maintaining notes is just another function call, no different in mechanism from calling a search tool or a calculator; what makes it "memory" is purely how the result is used, as persistent working state for the current task rather than as a one-off piece of information to act on and discard.

Notes versus long-term memory: a clear line

It is easy to blur notes and long-term memory together because both involve writing something down outside the immediate conversation. The distinction that matters is scope and lifespan. Notes are scoped to a task and are typically discarded, or archived without being actively used, once that task finishes; their job is to help the agent stay coherent across the many loop iterations a single complex task might need. Long-term memory is scoped to the user, the project, or the organisation, and is designed to outlive any single task, resurfacing in future, unrelated conversations. A research assistant's note that it has already read sources A and B for the report it is currently writing should probably vanish once the report is delivered. A fact that the user always wants citations in a particular format should probably live in long-term memory indefinitely, because it applies to every future report, not just this one.

Putting the three together: a worked scenario

It is worth walking through a single scenario end to end to see how short-term memory, long-term memory and notes interact, rather than treating them as three separate, unrelated topics. Consider a research assistant agent, similar in spirit to the project you will build in part 12, asked by a returning user to "Write me a short brief on recent developments in solid-state batteries, like last time but shorter."

1. Before the agent even starts planning, the system retrieves relevant long-term memories using the kind of similarity search described earlier: it finds a stored semantic memory noting that this user always wants sources cited as numbered footnotes, and an episodic memory recording that three weeks ago the agent produced a solid-state battery brief for this same user and the user said it was too long. Both get inserted into the prompt as context.
2. Reading the request and this retrieved context, the agent, following the decomposition habits from part 5, writes an initial plan to its notes: "Goal: short brief on solid-state batteries, cite as numbered footnotes, keep it shorter than last time (last brief was roughly 600 words, aim for 250). Steps: 1) search for recent developments, 2) select 3 to 4 strongest sources, 3) draft brief with footnotes, 4) check length."
3. The agent now enters the loop from part 2, calling a search tool and reading results. Each tool call and its result goes into short-term memory, the growing context window, exactly as described earlier in this article. After the search step, the agent updates its notes to a short summary of what it found, rather than keeping the full search results verbatim in its working notes, and could optionally also summarize the raw tool output out of the context window if the task runs long.
4. The agent drafts the brief, checks the word count against the 250-word target recorded in its own notes, and revises down if needed.
5. Once the brief is delivered, the task ends. The notes, having served their purpose, are discarded or archived without further use. But before ending, the system may write one new fact to long-term memory: "User requested a 250-word brief after being told the previous one was too long," an episodic memory that could usefully inform length choices the next time this user asks for a brief on any topic.

Notice how each kind of memory did a job the others could not. Long-term memory supplied continuity across sessions, letting the agent avoid repeating a known mistake from weeks earlier. Short-term memory, the context window, carried the detailed back-and-forth of this particular search and draft so the model could reason over it. Notes kept the agent anchored to its own plan and target length across several loop iterations, without needing to re-derive the goal from scratch or re-read the entire search transcript every time. None of these three would have been sufficient alone: short-term memory without long-term memory would repeat the same mistake every three weeks; long-term memory without notes would leave the agent without a running plan during execution; notes without long-term memory would forget the user's standing citation preference the moment the session ended.

Common mistakes

• Treating the context window as free long-term storage. Nothing in the context window survives the end of a session, no matter how large the window is; if information needs to persist, it has to be written somewhere outside the window on purpose.
• Storing every tool result and every message in a vector database without curation, which dilutes retrieval quality and makes it harder for genuinely important memories to surface above routine operational noise.
• Never updating or retiring stale memories, so the system confidently retrieves and acts on information the user corrected or that is simply no longer true, with no mechanism to notice the conflict.
• Confusing notes with long-term memory and either persisting a task-scoped scratchpad forever, where it accumulates and becomes irrelevant clutter, or discarding a genuinely durable fact as if it were task-scoped working state.
• Letting summarization run unsupervised on information that is still actively needed, compressing away a detail the next step in the task actually depends on, simply because it happened a few turns earlier.
• Skipping memory design entirely for a task that does not need it. Many agents run a single short task and finish; for these, short-term memory within one context window is entirely sufficient, and adding a vector database and a notes system adds complexity and failure points without adding value. The question from part 1, of whether you need an agent at all, has an analogue here: ask whether you need long-term memory at all before building it.
• Assuming retrieval by embedding similarity is infallible. It finds text that is semantically similar to the query, not necessarily the single most useful memory for the situation; a system that blindly inserts the top search result without any relevance check can occasionally pull in something misleading.

Summary and what comes next

An agent has no memory of its own between calls; everything it appears to remember is text that something deliberately placed back in front of it. Short-term memory is the context window itself, the running transcript of the current task, and it grows with every loop iteration unless it is actively managed with truncation, sliding windows or summarization. Long-term memory is information stored outside the model and retrieved on demand, often using embeddings and similarity search, and it is what lets an agent carry facts, preferences and past outcomes across entirely separate sessions, provided it is curated and kept up to date rather than allowed to accumulate unchecked. Notes, or a scratchpad, are a third, distinct kind of memory: task-scoped, agent-authored working state that keeps a long or multi-step task, of the kind introduced in part 5, coherent across many loop iterations without needing to re-read the full history or re-derive the plan each time.

Deciding what to keep in each of these places, and more specifically what to put into the context window at any given moment given everything that is available, is a skill in its own right, and that is exactly where part 7, Context Engineering, picks up. Where this article was about the three kinds of places information can live, the next one is about the craft of choosing, at each step of the loop, what actually belongs in front of the model right now.