AI memory explained — short-term vs long-term, and why the difference matters

"Add memory to the assistant" is one of those requests that sounds like a single feature and is actually two. When a product owner says memory, they usually mean the bot should remember what I told it. When an engineer hears it, they should be asking: remember it for the next three turns, or for the next three months? Those are different mechanisms, with different costs, different failure modes, and different code paths.

The word gets conflated because both feel like memory to the user. But short-term memory and long-term memory in an AI app share almost nothing under the hood. One is ephemeral and paid for by the token on every single call. The other is persisted, retrieved on demand, and survives across sessions. Designing them as if they were the same thing is how you end up with a bot that either forgets everything the moment a session ends, or drags a bloated, expensive context around until it collapses under its own weight.

The two tiers

Short-term memory is the context window. It's everything you pass into the model on a given call: the system prompt, the running conversation, any facts or documents you've injected, and the current user query. The model has no memory of its own between calls — it is stateless. What looks like "the model remembering the last message" is really your application resending that message in the next request. Short-term memory is whatever you choose to include in the payload this turn, and nothing more.

That has two consequences people underestimate. First, it's ephemeral: the instant the call returns, the model retains nothing. Second, it's expensive and finite. Every token in the context is billed on every call, and the window has a hard ceiling. Short-term memory is not free storage you dump things into — it's a budget you spend.

Long-term memory is a persisted store you retrieve from. It's the vector database, the structured records, the rolling summaries — knowledge that lives outside the model and outside any single request. It survives session boundaries, process restarts, and days of inactivity. But the model can't see it until you go and fetch the relevant pieces and place them into the context window. Long-term memory is only useful when it becomes short-term memory at the right moment.

If you borrow the cognitive-science analogy: the context window is working memory — the small, fast scratchpad of what you're actively thinking about. The persisted store splits into episodic memory (what happened — past conversations, events, decisions) and semantic memory (facts and knowledge — the user's preferences, your product docs, domain rules). The whole engineering problem is moving the right things between the persisted tiers and working memory, in both directions, at the right time.

Short-term memory is a budget, not a bucket

Treat the context window as a fixed-size bar that several things compete for. The system prompt takes a slice. The conversation history takes a slice that grows every turn. Retrieved long-term memory takes a slice. The current query takes a slice. And you must leave room for the output — the model can't respond if there's no budget left for the response.

The naive design just appends every turn to the history and resends the whole thing. It works beautifully in a demo and falls over in production. By turn forty, you're paying for forty turns of history on every call, latency has crept up, and you're brushing against the window limit. Worse, you hit the lost-in-the-middle problem: models attend most reliably to the start and end of a long context and reliably miss things buried in the middle. A fact from turn six can be technically present and functionally invisible.

So short-term memory has to be actively managed. You don't get to keep everything. You decide what stays verbatim, what gets compressed, and what gets evicted to long-term storage — which is exactly where the write path comes in.

The write path: deciding what's worth keeping

Not everything in a conversation deserves to be remembered. "What's the weather?" doesn't. "I'm on the enterprise plan and my team is in Berlin" does. The write path is the logic that decides what leaves the ephemeral context and earns a place in durable storage.

Extraction. Run over the conversation and pull out the durable facts — preferences, identities, decisions, commitments, constraints. This is usually a model call in its own right: given this exchange, list any stable facts about the user or the task worth remembering. You store the extracted facts, not the raw transcript. "User prefers metric units and dark mode" is worth a handful of tokens forever; the four messages it took to establish that are not.

Summarization and compaction. When the running history grows too long, compress the older portion into a summary and keep only the recent turns verbatim. The last few exchanges stay word-for-word because immediate coherence depends on them; everything older collapses into a paragraph of "here's what happened earlier." This is how you keep a long conversation coherent without paying for its full length on every call.

Persistence. Extracted facts land somewhere queryable. Semantic facts often go into a structured store — a plain table of key-value preferences, profile fields, settings — because you want exact, reliable lookups, not fuzzy similarity. Episodic content and unstructured knowledge go into a vector store, embedded so you can retrieve by meaning later. Use both. A vector store is the wrong tool for "what plan is this user on"; a SQL row is the wrong tool for "find past conversations that felt like this one."

The read path: getting it back into context

The read path runs at the start of (or during) a turn: figure out what the model needs to know right now, fetch it, and inject it into the context window.

Retrieval. For semantic facts, this is a keyed lookup — pull the user's profile and preferences by ID. For unstructured knowledge, it's a similarity search: embed the current query, find the nearest chunks in the vector store. This is the retrieval in RAG, and it's the same machinery whether you're recalling documents or recalling the user's own past.

Relevance and ranking. Retrieval returns candidates; you don't inject all of them. Top-k with a similarity threshold, often followed by a reranking pass, keeps you from stuffing the context with marginally-related material. Every irrelevant chunk you inject is budget spent and a small nudge toward the lost-in-the-middle problem. Precision beats recall here — a few highly relevant facts beat twenty vaguely related ones.

Injection. The retrieved memory gets formatted and placed into the context, usually as a labelled block: Known facts about this user: … / Relevant past context: …. Now it's short-term memory, visible to the model for this call, and the loop closes.

The loop

Put the two paths together and you get a cycle. A session begins, you read relevant long-term memory into the context, the model responds, you extract anything new worth keeping and write it back to the store, the ephemeral context is discarded, and the durable knowledge remains for next time. The context window is the working surface; the store is the ground truth. Facts flow up into working memory on demand and back down into storage when they've earned it.

Practical design decisions

What to keep in context vs. persist. Keep in context: the system prompt, the last N turns verbatim, and the specific facts retrieved for this task. Persist: stable user facts, decisions, and summaries of older history. If something is stable and reusable across sessions, it belongs in the store, not in an ever-growing transcript.

Pick a compaction trigger. Compact by token budget, not by turn count — summarize when the history slice exceeds, say, 40% of the window, so retrieval and output always have room. Keep the most recent turns raw and summarize the rest.

Match the store to the query. Structured store for exact lookups (preferences, plan, IDs). Vector store for "find things like this." Rolling summaries for narrative continuity. Most real systems use all three, each for what it's good at.

Cap the context, always. Set a hard budget for each slice — system, history, retrieved, query, reserved output — and enforce it. Unbounded context growth is the single most common way memory features quietly become slow and expensive.

Mistakes

Treating "add memory" as one feature. It's two systems — an ephemeral budget and a persisted store — with a write path and a read path connecting them. Scope them separately.

Letting context grow without bound. Appending every turn forever is the default that dies in production: rising cost, rising latency, and facts lost in the middle. Compact early.

Persisting raw transcripts instead of extracted facts. Storing whole conversations makes retrieval noisy and expensive. Extract the durable facts and store those; the transcript is not the memory.

Using a vector store for everything. Similarity search is the wrong tool for exact facts. "Which plan is this user on" should never be a fuzzy match.

Retrieving too much. Dumping twenty chunks into context to be safe spends budget and buries the signal. Rank hard, inject little.

Never forgetting. Long-term memory needs eviction and staleness rules too. A preference the user changed six months ago is worse than no memory — it's confidently wrong.

TL;DR

Memory in an AI app is two mechanisms, not one. Short-term memory is the context window: ephemeral, token-costed, finite, and lost the moment the call returns — manage it as a budget, not a bucket. Long-term memory is a persisted store — vector for similarity, structured for exact facts, summaries for continuity — that survives across sessions but is invisible until you retrieve it.

A write path decides what leaves the ephemeral context and earns durable storage: extract facts, summarize old history, persist to the right store. A read path pulls the relevant pieces back in: retrieve, rank, inject. Together they form a loop — read long-term memory into short-term context, respond, write new facts back out.

Design both explicitly, cap the context, match the store to the query, and forget on purpose. That's the difference between a bot that remembers you and one that either forgets everything or drowns in its own history.

If you're designing memory for an AI product, let's talk.