Librarian — Episodic Memory for AI Agents

A production system that gives AI coding agents a memory that survives across sessions. When a session ends, Librarian condenses it into a structured, embedded digest; later, an agent can ask “have I dealt with this before?” and get back the relevant past decision, fix, or dead-end — instead of starting blind every time.

1. Context & the problem

AI agents are stateless between sessions. Every new session starts blind: the agent has no idea what was decided yesterday, which approach already failed last week, or what state a long-running project is in.

Claude Code auto-writes a full transcript of every session to disk — but a raw transcript is the wrong artifact to remember from:

• It is too long to reload into a context window.
• It is too noisy to search — ninety percent is mechanics; the value is a few decisions buried inside.
• Document-style RAG answers “what does this document say”, which is a different question from “what did I already do about this.”

The cost of no memory is concrete: the agent repeats mistakes, re-litigates settled decisions, and loses continuity on anything spanning more than one sitting. I was paying — in time and tokens — to re-explain context I had already established.

The task I set myself: capture the signal from each session, make it semantically searchable, and expose it to any agent through a clean interface — without leaking secrets and without a frontier-model bill.

2. My role — stated honestly

I am not presenting myself as the person who hand-wrote every line of Node.js. My contribution is the part that is hard to outsource and hard to fake in an interview:

• I owned the problem framing and the architecture. Every entry in the decision log below is a judgment call I made and can defend.
• I directed the implementation through an AI coding agent — specifying behaviour, reviewing output, and rejecting designs that didn’t fit (the mid-project pivot in §6 is one example).
• I operate the system in production. I diagnosed and fixed a six-day silent data-capture gap (§7), and I maintain the runbook, the invariants, and the incident log.

This is an AI-native engineering model: the leverage is in judgment, decomposition, and operational ownership rather than in typing speed.

3. The core idea — intelligence at both ends

The design thesis, and the thing that separates Librarian from a naive “log everything and keyword-search it” store:

An LLM sits on the write path and an LLM sits on the read path — and that is deliberate.

• On write (input): raw transcripts are never stored as memory. An LLM (Gemini Flash) reads the session and produces a compact, structured digest — what we worked on / what problem / how it was solved / what’s left open — plus tags, a domain, an importance score, and a one-sentence summary. Signal extracted, noise discarded: recording everything indiscriminately produces a haystack, not a memory.
• On read (output): recall is not auto-injected into every prompt. It is a tool the consuming agent chooses to call, and the agent applies its own judgment to what comes back — deciding whether a recalled memory is relevant, current, and worth acting on.

The bet is that selectivity at both ends beats a dumb pipe in either direction — dump-everything on write, or blind-inject on read. The smart part is what gets remembered, and what gets recalled and used. I validate this by spot-checking recall against sessions I remember; an automated recall-quality harness (a golden query set) is the top open item — I do not yet measure precision numerically, and I say so plainly rather than assert it.

Everything below is in service of this thesis.

4. Architecture

Two independent services, cleanly split into a writer and a reader — a system can capture memory and serve memory without the two paths interfering.

Mac                              Raspberry Pi 500
───                              ────────────────
~/.claude/projects/*.jsonl  ──►  rsync mirror (cron, every 15 min)
 (Claude Code transcripts)                │
                                          ▼
                        [WRITE]  harvester  (systemd, polls every 5 min)
                                          │
                                          ├─► leak-check (regex redaction)
                                          ├─► Gemini Flash  → structured digest + tags + summary
                                          ├─► split digest into bullet-level chunks
                                          ├─► Vertex AI     → 768-dim embedding per chunk
                                          ├─► PostgreSQL    → memories (1/session) + indexed_chunks (many)
                                          └─► Telegram notification

                        [READ]   MCP server (HTTP/SSE, port 3201, Tailscale-only)
                                          ├─► search_chunks   (semantic search)
                                          ├─► get_digest      (full digest by id)
                                          └─► list_sessions   (chronological browse)
                                          │
                                          ▼
                                   Claude Code agents (call tools on demand)

Two-layer storage. One memories row per session holds the full human-readable digest; many indexed_chunks rows hold one embedding per bullet, each linked back to its parent. Retrieval ranks on the fine-grained chunks, then follows the link to the full digest only when needed — a cheap two-step over one expensive blob.

Three-level scope isolation. project (coarse) → domain (six fixed buckets: technical, business, reflections, core, content, private) → tags (open-ended). An agent can narrow before it searches, so recall is targeted rather than fuzzy — and private-domain memory stays isolated from shared agents.

MCP as the interface. Exposing recall through the Model Context Protocol means any MCP-compatible agent gets memory with zero custom glue — the same integration shape the agents already use for other tools.

Security posture (stated, not hidden). The recall endpoint has no application-level auth: the Tailscale network boundary is the perimeter, which is the right trade-off for single-operator use. Token-signing would be a prerequisite before any multi-user exposure — a deliberate, documented decision, not an oversight.

5. Selected engineering decisions

The full log has 11 entries; these four best show the reasoning style. Each was written down at the time with context, decision, and consequences.

Precision over recall — chunk by structure, not by sliding window. The industry-default RAG chunking is a ~500-character sliding window. Digests already have explicit structure (named sections, numbered bullets), so I chunk one bullet = one chunk instead. One concept per embedding raises ranking precision for narrow queries; a sliding window would have averaged unrelated topics into one vector and blurred recall. (Trade-off accepted: chunks vary in size and the chunker is digest-specific, not a general splitter.)

Model the memory, not a graph — an explicit non-goal. There is constant temptation to bolt on the latest graph-RAG paper (LightRAG, Mem0, etc.). I wrote an ADR drawing the line: those systems optimise recall over a static document corpus; Librarian optimises precision over personal session transcripts, where the priority is not hallucinating a decision I never made. The rule that came out of it — borrow retrieval-side mechanics (rerankers, query rewrite), never the storage model wholesale — turns every future “should we adopt X?” from a half-day comparison into a one-line check against a written non-goal.

Right-sized model choice. Digest extraction runs on Gemini Flash on the free tier, not a frontier model. Summarising a session doesn’t need frontier reasoning; the free tier covers extraction, and the only metered cost is Vertex AI embeddings (billed per character, cents/month at this volume). Right tool for the job, not the most expensive one — a cost decision I can justify.

Independence over coupling. A neighbouring MCP service had corrupted source on disk (see §7 lineage). Rather than block on repairing it, I made the writer talk directly to PostgreSQL and gave Librarian its own recall server. The system now lives and dies on its own code — no shared component can take it down.

6. A design pivot under real signal

The original MVP was a “Correction Hunter” — extract every “don’t do X, do Y” moment from a session and store it as a rule. First smoke tests killed it: the LLM over-generalised a single in-context remark into a blanket rule, and the same remark phrased differently produced duplicate rules that hashing couldn’t dedupe.

I flipped the output format mid-build: one factual digest per session instead of N prescriptive rules. Digests are fact-shaped (“what happened”), which is a far smaller hallucination surface than prescriptive rules (“what should always be done”), and one-record-per-session made idempotency trivial (hash(session_id) — one row by construction).

The lesson I take to interviews: I changed the core abstraction because the evidence said to, not because the original plan was sunk cost.

7. Results

• In production since 16 April 2026, used daily as the live memory layer for the several agents I run — this is not a demo. Roughly three months of continuous operation, interrupted only by the incident below.
• 550+ sessions digested · several thousand searchable digest chunks (5–15 per session), embedded and queried by cosine similarity across six domains — dominated by technical work with a steady tail of business and reflections. (These counts are the genuinely-digested-since-launch figures; they exclude ~4,150 pre-launch sessions marked skip and a separate legacy file-index corpus that recall filters out.)
• Marginal running cost — cents/month. Free-tier extraction model, self-hosted Postgres on a Raspberry Pi; the only metered line is per-character embeddings at low volume.
• In daily use it answers three recurring questions well enough that I rely on it: did we already solve this? · what was the state of this project? · what happened last time we touched this system?

A production incident I own. A deployment stripped the executable bit off a service wrapper; after the next reboot the harvester silently failed to start — a six-day gap in memory capture before I caught it. The fix was not a one-off patch but a set of durable guards I designed and documented as invariants:

• the service now re-asserts the executable bit on every start (ExecStartPre=chmod +x), so the failure mode cannot recur;
• restart-rate limits were corrected so a genuinely failing service actually enters the failed state instead of restart-looping forever;
• an OnFailure hook now fires a Telegram alert — so the next silent failure is loud;
• enable-linger guarantees user services come up on boot without a login.

Diagnosing a silent, delayed-onset production failure and converting it into codified invariants — rather than a quiet hotfix — is the operational maturity these roles look for.

8. Tradeoffs & lessons (honest notes)

• Summarisation is lossy by design. The digest is a lens, not a mirror; the summariser’s quality is the memory’s quality. Mitigation is one call away (get_digest), but a weak model yields weak memory.
• Taxonomy needs discipline. Domains and tags only help if applied consistently; left unmanaged they drift into noise. I enforce normalisation on the write path and seed the vocabulary into the extraction prompt — belt and braces.
• Semantic recall can mislead. Cosine similarity surfaces things that sound related but aren’t. Recall is treated as a starting point for the agent’s judgment, not a verdict — which is exactly why the read path keeps an LLM in the loop (§3).
• Recency vs. relevance is a real tension. The most similar memory isn’t always the most current one; a stale digest can describe a state that no longer exists. So recall is context to verify against live reality, never truth to act on blindly.

9. What this demonstrates

Built and operated by Slava Spitsyn. Source is private (personal infrastructure); this write-up describes the system without exposing secrets, credentials, or private content.