Self-hosted AI memory systems

Objectives

• Give a self-hosted agent memory that survives restarts and stays useful over months.
• Separate kinds of memory (working context, episodes, facts and preferences) instead of one undifferentiated pile.
• Compare keyword search, vector search and a combination of both on real questions.
• Keep everything inspectable: the user should be able to see, correct and delete what is stored.

Architecture

The sketch has four parts. A small working memory holds recent turns. A consolidator runs on a schedule, distils those turns into durable facts with a pointer to their source, and writes them to a long-term store. A retriever answers the agent's lookups with hybrid search, combining full-text and vector results.

Implementation notes

Background reading and the first schema are in the journal article Exploring Memory Systems for Self-hosted AI Agents. Current direction:

• Start with SQLite and its FTS5 full-text extension. It runs anywhere, needs no extra service, and its BM25 ranking is easy to reason about.
• Add embeddings only where keyword search demonstrably misses, and combine the two result lists with reciprocal rank fusion.
• Every stored fact keeps its source and a last_used_at timestamp, so decay and auditing are possible later.

Results

Sample entry. This experiment is still exploring. Record findings as they arrive, ideally against a small, fixed set of real questions with known answers, so changes can be compared fairly.

Limitations

• Consolidation depends on a model to extract facts, and models can extract wrong ones. Source pointers make mistakes traceable, not impossible.
• There is no settled rule yet for contradictions between old and new facts.
• Embedding models change. Switching models means re-embedding everything.

Future improvements

• A review screen that shows what the agent learned this week, with approve and delete actions.
• Decay rules per kind of memory, tuned against real usage.
• Encryption at rest for the memory database.