Exploring Memory Systems for Self-hosted AI Agents

An agent without memory starts every conversation as a stranger. An agent that remembers everything becomes slow, expensive and occasionally wrong about things that stopped being true months ago. These are working notes on the space between: what to keep, where to keep it, and how to find it again when it matters.

They are notes, not conclusions. The AI Lab entry on memory systems tracks the experiment itself.

Four kinds of memory

Research on language agents often borrows terms from cognitive science. They are useful because each kind wants a different store:

• Working memory is what the model sees right now: the recent turns of the conversation. It is limited by the context window and it is the most expensive memory per token.
• Episodic memory is what happened: past conversations and the outcomes of tasks.
• Semantic memory is what is true: facts and preferences, such as "deploys go out on Thursdays".
• Procedural memory is how to do things: reusable skills and instructions. Hermes Agent's skills fall into this category.

Most early designs put everything into one vector index and hope retrieval sorts it out. Separating the kinds makes it much easier to decide what to forget.

Start with plain text search

Vector search gets the attention, but keyword search is cheap, predictable and surprisingly strong for the things people actually ask an assistant about, such as names, hosts, error messages and dates. SQLite ships it as the FTS5 extension, with BM25 ranking built in.

CREATE TABLE memories (
    id           INTEGER PRIMARY KEY,
    kind         TEXT NOT NULL CHECK (kind IN ('fact', 'episode', 'preference')),
    content      TEXT NOT NULL,
    source       TEXT,
    created_at   TEXT NOT NULL DEFAULT (datetime('now')),
    last_used_at TEXT
);

CREATE VIRTUAL TABLE memories_fts USING fts5(
    content,
    content='memories',
    content_rowid='id'
);

CREATE TRIGGER memories_ai AFTER INSERT ON memories BEGIN
    INSERT INTO memories_fts(rowid, content) VALUES (new.id, new.content);
END;

CREATE TRIGGER memories_ad AFTER DELETE ON memories BEGIN
    INSERT INTO memories_fts(memories_fts, rowid, content) VALUES ('delete', old.id, old.content);
END;

CREATE TRIGGER memories_au AFTER UPDATE ON memories BEGIN
    INSERT INTO memories_fts(memories_fts, rowid, content) VALUES ('delete', old.id, old.content);
    INSERT INTO memories_fts(rowid, content) VALUES (new.id, new.content);
END;

The full-text index points at the main table instead of copying it, and the triggers keep the two in step. Querying it is one join:

SELECT m.id, m.kind, m.content, bm25(memories_fts) AS score
  FROM memories_fts
  JOIN memories AS m ON m.id = memories_fts.rowid
 WHERE memories_fts MATCH ?
 ORDER BY score
 LIMIT 8;

BM25 scores are lower for better matches, which is why the sort is ascending.

Add vectors where words fail

Keyword search misses paraphrases. "The server keeps running out of RAM" will not find a memory about "OOM kills on the VPS". Embeddings close that gap. The question is how to combine the two result lists without tuning weights forever.

Reciprocal rank fusion is a simple answer. Each list contributes a score based only on rank, so the two scoring scales never need to agree:

def reciprocal_rank_fusion(*rankings: list[int], k: int = 60) -> list[int]:
    scores: dict[int, float] = {}
    for ranking in rankings:
        for rank, memory_id in enumerate(ranking, start=1):
            scores[memory_id] = scores.get(memory_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

A memory that ranks well in both lists rises to the top. One that only one method found still gets a chance.

Consolidate instead of hoarding

Storing every turn verbatim is easy and quickly becomes noise. A better pattern is a consolidation pass that runs on a schedule:

1. Read the conversations since the last pass.
2. Ask the model to extract durable facts and preferences, each with a pointer to where it came from.
3. Compare each new fact with existing ones and update, merge or mark old facts as superseded.
4. Keep the raw episode for a while, then let it expire.

The pointer back to the source is what keeps this honest. When the agent states a fact, you can ask where it learned it.

Questions still open

• Contradictions. When two memories disagree, newer is not always right. Recording a confidence and a source for each fact helps, but the rules are not settled.
• Forgetting. A last_used_at column makes decay possible. How aggressive decay should be depends on the kind of memory, and the right curve needs real usage to tune.
• Privacy. Memory is personal data by definition. It needs encryption at rest, a way to show the user what is stored, and a delete button that really deletes.

The direction so far is clear enough: separate the kinds of memory, start with search you can reason about, and treat consolidation as a first-class job rather than an afterthought.