All research systems
MemKitActive

How should long-lived agents retain useful information without memory becoming unbounded, stale, or contradictory?

Question → hypothesis/design → architecture → evaluation → results → limitations → next questions

Motivation

Most agent 'memory' is vector search over chat history. That is context management, not memory. 'I love my job' and 'I quit' both mention the job and retrieve together on similarity alone, so a naive agent synthesizes both into an answer that's wrong the moment it's assembled. Similarity is not truth. MemKit's starting bet is that solving that problem comes before optimizing recall quality, not after. The current build deliberately has no semantic search yet, because a fast wrong answer is worse than a slightly slower right one.

Hypothesis

Explicit, write-time conflict classification, resolving whether a new fact adds, updates, or duplicates what's already known, will remove stale and contradictory retrievals that naive history replay and similarity-only retrieval structurally cannot catch, at a write-path cost worth paying.

Threat model

  • Staleness: a fact that was true last month outranks the fact that replaced it, because similarity alone has no notion of time or truth
  • Contradiction: two incompatible facts coexist in the store and retrieval returns whichever is closer to the query, not whichever is current
  • Lexical blindness: 'I love my job' and 'I hate my job' share almost no overlapping tokens, exactly where a cheap heuristic goes blind
  • Unbounded growth: superseded facts accumulating forever, even once they're known to be stale
  • Availability: a memory layer that depends on an external model call to resolve every write is a memory layer that goes down when a model provider does

Architecture

  • A single flat fact store, not a multi-tier memory model. Every fact is either active or archived-as-superseded, with no separate episodic/semantic/symbolic layers
  • conflict-lens: a dependency-free classification module deciding add / update (supersede) / duplicate for each new fact against what's already known
  • Cheap token-overlap heuristic handles the clear cases; an optional small-model resolver is consulted only for the genuinely ambiguous band
  • Superseded facts are archived, not deleted, visible as history but excluded from active recall
  • Retrieval today is lexical plus time-decay, not embedding-based. Semantic recall is explicitly not yet built

Memory model

  • A write is classified before it's stored: add (genuinely new), update (supersedes an existing fact), or duplicate (already known)
  • The heuristic is free and handles most cases; the model-backed resolver is the expensive path, used only when token overlap can't decide
  • The resolver's system prompt is cached and it uses a small, fast model, so the ambiguous band is the only place cost is spent
  • With the Claude resolver, a failure falls back to the heuristic so the write is never blocked. In the open local-resolver pull request, a failed or invalid judgment adds the fact and supersedes nothing
  • A background job prunes archived facts past a configurable retention window, independent of the read-time decay used in ranking

Implementation

  • Single Go binary, SQLite-backed: no external services, no Python runtime, one process you own
  • REST API: write, search, explicit supersede, hard delete, and a GDPR-style per-user erasure endpoint
  • conflict-lens factored out as its own standalone, dependency-free module, reusable outside MemKit, one tested implementation rather than two
  • memkit-mcp: a dependency-free stdio MCP bridge, so any MCP client can use MemKit as long-term memory without talking REST directly
  • Multi-tenant via a per-tenant identifier; all data additionally scoped by user id

Evaluation methodology

M2 has run on a held-out workload (see results below), against a real MemKit instance and real baseline implementations, after an earlier published result failed to reproduce. M4 has partly run. M1 and M3 are still the planned design, kept here so the plan stays checkable against what eventually gets published, not retrofitted after the fact.

Workloads
  • Multi-week operational assistant: recurring entities, facts that change over time
  • Long incident thread: high-volume, low-signal history with a few decisive facts
  • Contradiction workload: facts deliberately updated and reversed across sessions
Baselines
  • Naive conversation-history retrieval
  • Vector-only retrieval (embeddings + top-k)
Metrics
  • Retrieval precision and recall against labelled gold memories
  • Stale-retrieval rate and contradiction rate per 100 queries
  • Context tokens consumed per task
  • Storage growth per 1k interactions
  • Retrieval and write-path latency (p50 / p95)

Experiments

Planned M1: Retrieval quality

Labelled gold memories across three workloads (multi-week assistant, long incident thread, contradiction workload) queried under MemKit, naive history retrieval, and vector-only top-k.

Measures Precision, recall, and answer relevance under a fixed context budget. Not yet built. No gold labels exist yet.

M2: Staleness and contradiction

Facts updated, reworded, restated, shared between different people, and added alongside other attributes; queries ask for the current value or for a fact that should have survived.

Measures Stale rate, and the rate at which a fact that should survive is missing from the top five results, per category and approach. Run: a held-out synthetic workload (development and test splits, 102 queries per category), heuristic only vs a local-model resolver vs vector retrieval. See results below.

Planned M3: Growth and cost

Identical interaction streams replayed to each approach over a simulated multi-week window.

Measures Storage growth, context tokens consumed per task, write-path latency and cost. Not yet run.

M4: Resolver ablation (partly run)

Heuristic only versus a local Ollama judge (llama3.1:8b) on the held-out workload. The Claude-based resolver was not evaluated.

Measures Reported under results: lost facts fell sharply, stale answers did not, at a median 4.6 s per write.

Results

Preliminary results
  • Correction: an earlier version of this page reported 52.0% stale and 51.7% contradiction on a 400-query workload. That result does not reproduce. On a clean database the current code gets 21% stale (95% interval 17% to 25%), and replaying the same user's workload a second time on the same database gives 28%, so the original run most likely accumulated state across repeated runs. The vector-similarity baseline did reproduce (80.5% stale)
  • On a held-out workload (synthetic; development and test splits with disjoint attributes, names and values; 102 queries per category; one run per configuration), the heuristic alone made another person's fact missing from the top five results in 71.6% of test cases, and another attribute of the same person in 50.0%, because it counts a shared name as word overlap
  • A local-model judge (Ollama, llama3.1:8b, top ten candidates) cut those to 7.8% and 3.9% on the test split, at a median 4.6 s per write against 1.3 ms for the heuristic. It did not reduce stale answers in general: 29.4% (same-frame updates) and 28.4% (reworded updates), against 15.7% and 33.3% for the heuristic, because the right fact is often not among the ten most word-similar candidates
  • Not re-verified: the write-path cost (575 ms median on real operational data, n=11) and the mem0 latency comparison from the original run have not been re-measured here and should be treated as unverified

Synthetic workloads only (fictional entities), one run per configuration, development iterated four times, and one text-only diagnostic was computed on the test split before settings were frozen. The local-model resolver and the evaluation harness are in an open pull request (github.com/agent-rails/memkit/pull/2), which holds the full protocol and disclosures. Marked preliminary.

Failure cases and risks — not an exhaustive catalogue

  • With the heuristic alone, a new fact about one person can supersede another person's fact that shares a name or value: that fact was missing from the top five results in 71.6% of held-out test cases
  • A new fact about one attribute can supersede a different attribute of the same person (50.0% of held-out test cases)
  • Even with a local-model judge, the right fact is often not among the ten most word-similar candidates (85% for same-frame updates and 69% for reworded ones on the test split), so stale answers remain
  • Two long sentences about different people that differ by one name can reach the duplicate threshold and skip the judge; the evaluation does not cover this
  • The heuristic alone cannot see a lexically-dissimilar contradiction unless the resolver is enabled and the case falls in its ambiguous band

Limitations

  • The held-out evaluation is synthetic, uses one local model and one run per configuration, and was iterated four times on its development split; the failure cases listed are observed on it and are not an exhaustive error analysis
  • No semantic or embedding-based recall yet. Retrieval today is lexical plus time-decay, a real and current gap, not a hypothetical one
  • SQLite only; no shared/concurrent-writer backend yet, which bounds this to single-agent or low-concurrency use for now
  • Conflict resolution assumes a single authoritative writer per fact domain, and multi-writer conflict has no defined resolution yet
  • The local-model resolver is in an open pull request and adds seconds of write latency. The Claude-based resolver was not evaluated, and its prompt has no check that two facts are about the same subject

Current status

v0.1: REST API, SQLite storage, conflict-lens (heuristic plus optional model-backed resolver for ambiguous cases) shipped and in use. Consolidation pruning and the MCP bridge are both live. A held-out evaluation has run and corrected an earlier published result (see results below). A search fix has merged; a local-model resolver and the evaluation harness are in an open pull request.

Roadmap

  • Postgres backend for shared/concurrent use
  • gRPC interface
  • Embedding-backed recall

Next questions

  • Whether structured conflict resolution generalizes past single-writer, single-tenant workloads
  • Multi-agent shared memory with conflict resolution across writers
  • Memory as a causal model rather than a retrieval problem