AI Agent Long-Term Memory Setup SOP: Making Agents Remember You Across Sessions

You spend half an hour with an AI assistant, laying out the project background, tech stack, naming conventions, and personal preferences. The next day you open a fresh session and it has forgotten all of it. You have to re-explain everything from scratch. That is the default state of most agents in 2026: close the context window, memory zeroes out. Short sessions are fine, but the moment you need collaboration that spans days or projects, there is no path forward without long-term memory.

This article walks through the practical process of wiring long-term memory into an agent: why agents forget, how to integrate mem0, how to integrate Letta, how to choose between them, and when you should reach for long-term memory instead of RAG or just leaning on the context window. Every step ships real commands and a copyable prompt, with a pitfall log and FAQ at the end. It pairs with our agent memory tools comparison: that piece covers how to pick, this one covers how to actually build it once you have picked. If you are still in the evaluation phase, you can also check the RAG evaluation SOP to quantify retrieval quality, or cross-reference the Dify knowledge base RAG build SOP to understand the RAG route.

Why Agents Forget in the First Place

Forgetting is not a sign the model is dumb. It is a physical constraint of the context window. Even with windows pushing past a million tokens, three problems are unavoidable:

• Windows are volatile: one session ends, the window clears, and the next conversation starts from zero.
• Windows are finite: long conversations pile up, retrieval precision drops, truncation kicks in, and key facts get pushed out.
• Windows do not separate speakers: user preferences, factual conclusions, and idle chitchat all blur together, so you cannot recall selectively.

Long-term memory tackles exactly these three things: extract the facts worth keeping, store them outside the window, and retrieve them on demand in the next conversation. It is not a replacement for the context window. It is a persistent storage layer sitting outside the window that lets information survive across sessions. The litmus test is simple: will I need this fact in the next session? If yes, write it to memory; if no, leave it in the window and let it expire.

Hands-on with mem0: A Memory Layer in Five Minutes

mem0 (pronounced mem-zero) is the most mainstream agent memory library today, with 62K GitHub stars, an Apache-2.0 license, and YC S24 backing. Its pitch is a universal memory layer for AI agents, and the core idea is to use an LLM to automatically extract facts from conversations, persist them into a vector store, and recall them by semantic search next time. In April 2026 it shipped a new algorithm that lifted its LongMemEval score from 67.8 to 94.4, with a single retrieval pass costing roughly 6.8K tokens.

Install and minimal working example

# Install the library (Python)
pip install mem0ai

# For hybrid search (BM25 keyword + entity extraction), add the nlp extra
pip install mem0ai[nlp]
python -m spacy download en_core_web_sm

By default mem0 uses OpenAI's gpt-5-mini for extraction and text-embedding-3-small for vectorization, so set OPENAI_API_KEY first. To swap in other models, see the official LLM and embedder configuration docs.

from openai import OpenAI
from mem0 import Memory

openai_client = OpenAI()
memory = Memory()  # default local config, user_id isolates users

def chat_with_memories(message: str, user_id: str = "default_user") -> str:
    # 1. Retrieve relevant memories for the current question
    relevant = memory.search(
        query=message, filters={"user_id": user_id}, top_k=3
    )
    memories_str = "\n".join(
        f"- {entry['memory']}" for entry in relevant["results"]
    )

    # 2. Inject memories into the system prompt
    system_prompt = (
        "You are a helpful AI. Answer based on query and memories.\n"
        f"User Memories:\n{memories_str}"
    )
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": message},
    ]
    response = openai_client.chat.completions.create(
        model="gpt-5-mini", messages=messages
    )
    assistant_response = response.choices[0].message.content

    # 3. Write this turn back to memory (mem0 extracts facts automatically)
    messages.append({"role": "assistant", "content": assistant_response})
    memory.add(messages, user_id=user_id)

    return assistant_response

Three APIs are the whole story: add writes memory, search retrieves, and user_id isolates users. memory.add() does not dump the raw conversation into the store. It runs an internal LLM call to extract facts ("user prefers dark mode", "project uses TypeScript") and stores those. That is why it beats dumping raw chat logs: you persist distilled facts, not verbose originals, and retrieval token cost stays under 7K.

The April 2026 algorithm brought a few changes worth knowing. First, ADD-only mode: memories only accumulate, never overwritten, with a single LLM call for extraction and no UPDATE/DELETE. Second, entity linking: extracted entities get embedded and linked across memories so retrieval can follow the thread. Third, multi-signal fusion: semantic vectors, BM25 keyword matching, and entity matching run in parallel and fuse scores, beating pure semantic search. Fourth, temporal awareness: a query about "what we use now" versus "what we used last month" returns different dated memory instances.

Retrieval tuning: do not ship the defaults

memory.search() defaults to top_k=3, returning only the three most relevant hits. Three knobs matter: top_k sets recall volume (3 to 10, more means more noise, less means missed hits); filters scopes by user, session, or agent; and hybrid retrieval requires the mem0ai[nlp] extra, otherwise you get semantic search only. After the first batch of memories lands, run a small "question, expected hit" test set and tune top_k by recall rate instead of guessing. For how to quantify retrieval quality, borrow the test-set approach from our RAG evaluation SOP and treat "question, expected memory" pairs as your eval set.

Self-hosted vs managed cloud

One command brings up the full self-hosted stack (dashboard, API, vector store), defaulting to http://localhost:3000. Teams with sensitive data that cannot leave the network go self-hosted. To get going fastest, the pip library is enough.

CLI for quick validation

Skip the code and validate the flow from the terminal first:

npm install -g @mem0/cli   # or: pip install mem0-cli

mem0 add "Prefers dark mode and vim keybindings" --user-id alice
mem0 search "What does Alice prefer?" --user-id alice

Hands-on with Letta: A Stateful Agent with Memory

Letta (formerly MemGPT) takes a different route. Instead of bolting a memory layer onto an agent, it hands you a stateful agent runtime. It has 24K GitHub stars and an Apache-2.0 license. The core difference: Letta lets the agent manage its own memory. What to remember, what to forget, when to page through the archive are decisions the agent makes internally, closer to a self-improving agent. mem0 is "you store and fetch for the agent"; Letta is "the agent stores and fetches for itself".

Install and spin up an agent

Letta recommends the new Letta Code CLI (requires Node.js 22.19+):

npm install -g @letta-ai/letta-code
letta   # launch a memory-enabled agent right in your terminal

To embed an agent inside your own app, use the Agent SDK:

npm install @letta-ai/letta-agent-sdk

import { LettaAgentClient } from "@letta-ai/letta-agent-sdk";

const client = new LettaAgentClient({
  backend: "cloud",              // or "local" for fully local
  apiKey: process.env.LETTA_API_KEY,
});

// Create a memory-enabled agent with a model and persona
const agentId = await client.createAgent({
  model: "anthropic/claude-opus-4-8",
  human: "Name: Alice. Role: PM.",
  persona: "I am a helpful assistant that remembers context.",
});

// Send a message and stream the response
await using session = client.resumeSession(agentId);
await session.send("What do you know about me?");
for await (const message of session.stream()) {
  if (message.type === "assistant") console.log(message.content);
}

backend: "local" runs the agent entirely on your machine with no cloud hop. backend: "cloud" routes through Letta's Constellation. The model is not locked in: Anthropic, OpenAI, and zAI all work, and the official model leaderboard has recommendations. If you are on the legacy V1 API, Python uses letta-client and TypeScript uses @letta-ai/letta-client, but for new projects the official recommendation is to go straight to the Agent SDK.

Letta's memory architecture differs from mem0 in philosophy. It gives the agent a memory block that the agent reads from and writes to like managing its own notebook. The upside is the agent can decide what is worth remembering and when to dig up old context, which suits long-horizon planning and self-improvement tasks. The cost is weaker control for you: it is harder to debug exactly what it remembered. If you want precise, controllable memory access, mem0 is the smoother fit. If you want to hand an agent a task and let it grow on its own, Letta fits better.

A prompt for memory extraction

Whichever tool you use, memory extraction quality comes down to the prompt. Here is a copyable template that has the LLM distill structured memories from a conversation:

You are a memory extractor. Read the conversation below and pull out facts worth remembering long-term.
Rules:
1. Extract facts only, not small talk or process filler ("hi", "let me check" are out);
2. One short sentence per memory, with a clear subject (user preference / project fact / decision made / to-do);
3. Distinguish preference from fact, prefix each with [preference] or [fact];
4. Deduplicate: do not re-extract anything semantically identical to a known memory;
5. Timestamp: where a date can be inferred, include it.
Output: one memory per line, no numbering, no explanation.
Conversation:
{conversation}
Known memories (for dedup):
{existing_memories}

Feed the extracted memories into memory.add() or into Letta's store. Rule 4, dedup, is the one that matters most. Skip it and the memory store bloats fast, and retrieval noise climbs in lockstep.

mem0 vs Letta vs Pure RAG: How to Choose

These three are not mutually exclusive. They sit at different layers. The table below compares them along real deployment dimensions:

One line: Got an existing agent and want to add memory, pick mem0. Building a long-horizon self-improving agent from scratch, pick Letta. Feeding static docs like product manuals and company policies, pick RAG. Many production agents want both: mem0 or Letta for interaction memory, RAG for external knowledge.

When to Use Long-Term Memory vs RAG vs the Context Window

This is the most commonly confused decision point. Here is a framework:

• Context window: information used within the current session, needed right now, disposable after. Example: a code snippet the user pasted in this turn. Lowest cost, no integration needed.
• RAG: external static knowledge, things the model does not know but can look up. Example: product manuals, API docs, company policies. The source is documents, update frequency is low, and retrieval is chunk-based. See our RAG evaluation SOP.
• Long-term memory: cross-session user-state, things the model should remember but loses when the window closes. Example: user preferences, past decisions, project background. The source is the conversation itself, and it updates continuously.

The mnemonic: Ask "will I need this next time?" If no, leave it in the window. If yes, ask "is it external knowledge or the user's own business?" External knowledge goes to RAG, the user's own business goes to long-term memory. The three stack into a complete picture: the window handles the present, RAG handles external knowledge, and long-term memory handles user context.

Pitfall Quick Reference

Pitfall 1: Memory store bloats with no cleanup. mem0's new algorithm is ADD-only, so memories accumulate and are never overwritten. Run it long enough and the store fills with semantically duplicate stale memories, dragging down retrieval precision and latency. Fix: run a dedup and merge pass on a schedule, set an expiry policy, and trigger a memory refresh when key facts change. Give the store a weekly health check, do not wait until retrieval is visibly slow.

Pitfall 2: Storing chitchat as memory. A weak extraction prompt lets "hi" and "got it" slip into the store. Fix: use the extraction template above, explicitly extract facts only, add the [preference] / [fact] prefix, and force dedup.

Pitfall 3: No user isolation for memory. In multi-user scenarios, forgetting to pass user_id leaks user A's preferences into user B's session. Both mem0 and Letta support user or agent isolation, but neither forces it by default. Before going live, make sure every add and search carries the user_id or the right isolation field.

Pitfall 4: Treating long-term memory like RAG. Stuffing hundreds of product docs into mem0 and using it as a knowledge base. The result is slow, inaccurate retrieval, because a memory store is designed for high-frequency read-write interaction memory, not bulk static document retrieval. Static knowledge belongs in RAG. Do not make the memory store do a job it is bad at.

Pitfall 5: Self-hosted without a default LLM configured. mem0 calls OpenAI by default locally. If OPENAI_API_KEY is missing or wrong, memory.add() fails silently (no error, but nothing gets stored). Fix: after the first run, manually call memory.search() to verify data actually landed. Do not assume success.

FAQ

Q: Do I have to pick exactly one of mem0 and Letta? A: No, but usually a project leads with one. mem0 is a memory layer you bolt onto an existing agent. Letta is an agent runtime that ships with memory built in. If you already have an agent and want memory, mem0 is lighter. If you are building a long-horizon agent from scratch, Letta saves you more work. Both self-host, so data stays in-house.

Q: Memory is stored, but how do I know retrieval is any good? A: Borrow the approach from our RAG evaluation SOP: build a set of "question, expected memory" test pairs and run hit rate. mem0 also open-sources its memory-benchmarks eval framework, which reproduces its LongMemEval scores. Do not rely on a gut feeling that "it seems to remember." Put a metric on it.

Q: Can I use models other than OpenAI? A: Yes. mem0 supports multiple LLMs and embedders, configured in the official docs. Letta is fully model-agnostic: Anthropic, OpenAI, and zAI all work. Swapping models mainly affects extraction quality and cost, not the store and fetch logic. In production, start with a cheap model to get it working, then upgrade based on results.

Q: Will concurrent writes corrupt memory? A: The self-hosted server and cloud platform both have concurrency control and API auth. The plain pip library does not, so under high concurrency you need your own locking or the self-hosted server. Letta's runtime processes messages for a single agent serially, so concurrency is less of a concern. When multiple agents share a memory store, always go through an authenticated server endpoint, do not let multiple processes write directly to the same local store.

Q: What is the shortest path for a first-timer? A: pip install mem0ai, set OPENAI_API_KEY, copy the chat_with_memories function above, and run a demo that can store and recall. Verify with memory.search() that it actually retrieves what the previous turn stored. Only then think about swapping models, self-hosting, or tuning the extraction prompt. Get it working first, then add features. Do not start by self-hosting the full stack.

参考来源

• mem0 repo and docs: https://github.com/mem0ai/mem0
• mem0 official docs: https://docs.mem0.ai
• mem0 paper (arXiv:2504.19413): https://mem0.ai/research
• Letta repo (formerly MemGPT): https://github.com/letta-ai/letta
• Letta official docs: https://docs.letta.com
• Related on this site: agent memory tools comparison | RAG evaluation SOP | Dify knowledge base RAG build SOP