The Memory Stack

Table of contents

Memory is a stack, not a switch

For a while I treated memory as a feature: flip it on, the agent remembers, done. It kept biting me. Memory was too sticky, too forgetful, or the wrong shape — a transcript when I wanted a fact, a fact when I wanted a timeline.

The fix: stop treating memory as one component. It’s a stack. Four layers, bottom to top:

Each layer only knows the one beneath it. The provider doesn’t know your rules. The MCP doesn’t care which provider it writes to. That separation is the point: swap a provider without touching a rule.

Layer 1: The integration provider

Where memory physically lives, plus the integration that reads and writes it. Your storage engine. Like databases, providers specialize — there’s no “best” one, just the one whose strengths match your workload.

• Mnemosyne — persistence. Durable memory that survives everything. Your floor.
• Built-in memory — the fast path. Always there, no setup, fine for the working set.

Don’t pick on vibes. Evaluate. LongMemEval and LOCOMO are the long-horizon benchmarks; trace runs with Langfuse to see what got recalled and what got dropped. Pick on numbers, not a landing page.

The provider is dumb storage plus retrieval. It doesn’t know what’s worth remembering. That judgment lives higher up.

Layer 2: The policy MCP

MCP (Model Context Protocol) is how the agent talks to memory. A policy MCP sits between the agent and the providers and decides how memory gets shaped, written, and searched. Same providers underneath, different memory depending on what’s on top.

The one I keep coming back to is mempalace. It doesn’t store blobs — it records entities, in a literal memory-palace structure:

• wings → rooms → drawers — a place for everything, so retrieval is structural, not fuzzy search
• tunnels — associative links, so you traverse “what’s related” instead of re-querying
• knowledge graph + taxonomy — entities, relationships, types you can query directly
• dedup on write — checks for duplicates before filing, so memory doesn’t bloat

Add session search — recall across past sessions — and you have what you need: write structured memories, find them across time, follow the links between them.

The MCP is policy plus structure. It turns “store this string” into “file this entity in the right drawer, link it, don’t duplicate it.”

Daily memory vs categorized memory

This split is why you run more than one thing.

• Daily memory is time-indexed. A diary. “What happened today?” mempalace’s diary and the graph timeline. For narrative continuity — picking up where you left off.
• Categorized memory is entity-indexed. A filing cabinet. “Everything I know about X.” Drawers, taxonomy, graph entities. For durable facts that don’t belong to any one day.

You don’t pick one. You compose: daily memory for what happened, categorized memory for what’s true, a persistence provider holding both up.

Layer 3: The memory law

The rules — what gets remembered, when, how it’s categorized. No provider, no MCP encodes your judgment. You do, in plain markdown.

The hierarchy I run, in precedence order:

1. .hermes.md / HERMES.md
2. AGENTS.md
3. CLAUDE.md
4. editor rules (e.g. cursor rules)

(SOUL.md is special — read from HERMES_HOME, not the project.)

These files hold the laws every layer below obeys:

• one fact per memory, not a transcript
• save preferences and decisions, not the conversation that produced them
• convert relative dates to absolute before filing
• don’t store what the repo already records

They’re laws because everything underneath follows them. The MCP files what they tell it to. The provider just stores it.

When the rules don’t fit, fork them — add your own, override a built-in. The law is the one part of the stack you’re meant to edit. But a law only governs tools that already exist. New tools live one layer up.

Layer 4: The memory router

A law governs the tools that exist. The router decides which tools exist.

It’s the memory code in the harness — the part that turns “remember this” into an actual operation. By default it does one thing: append to memory.md. That’s the floor.

Fork it. Modify the source, give the router the memory tools your work needs — a custom write path, a sharper recall query, a tool that files straight into the provider or MCP instead of a flat file. Different needs, different tools.

Providers and MCPs are interfaces you adopt. Laws are text you own. The router is code you rewrite.

Putting it together

Bottom-up, the whole thing is small:

• Provider — where memory lives. Pick by eval.
• MCP — how it’s organized. Structure, links, dedup.
• Law — what gets kept. Rules you wrote.
• Router — what’s possible at all. Source you fork when the rest runs out.

Memory isn’t a switch. It’s a stack — and a stack tells you where to look when it breaks. Too sticky? The law. Bad recall? The provider. Wrong shape? The MCP. Tool doesn’t exist? The router. One fuzzy “memory” feature, and you’re guessing. Four layers, and every problem has a floor.