All research systems
MemKitActive

How should long-lived agents retain useful information without memory becoming unbounded, stale, or contradictory?

Motivation

Most agent 'memory' is vector search over chat history. That is context management, not memory. 'I love my job' and 'I quit' both mention the job and retrieve together on similarity alone -- a naive agent synthesizes both into an answer that's wrong the moment it's assembled. Similarity is not truth. MemKit's starting bet is that solving that problem comes before optimizing recall quality, not after -- the current build deliberately has no semantic search yet, because a fast wrong answer is worse than a slightly slower right one.

Hypothesis

Explicit, write-time conflict classification -- resolving whether a new fact adds, updates, or duplicates what's already known -- will remove stale and contradictory retrievals that naive history replay and similarity-only retrieval structurally cannot catch, at a write-path cost worth paying.

Threat model

  • Staleness: a fact that was true last month outranks the fact that replaced it, because similarity alone has no notion of time or truth
  • Contradiction: two incompatible facts coexist in the store and retrieval returns whichever is closer to the query, not whichever is current
  • Lexical blindness: 'I love my job' and 'I hate my job' share almost no overlapping tokens, exactly where a cheap heuristic goes blind
  • Unbounded growth: superseded facts accumulating forever, even once they're known to be stale
  • Availability: a memory layer that depends on an external model call to resolve every write is a memory layer that goes down when a model provider does

Architecture

  • A single flat fact store, not a multi-tier memory model -- every fact is either active or archived-as-superseded, no separate episodic/semantic/symbolic layers
  • conflict-lens: a dependency-free classification module deciding add / update (supersede) / duplicate for each new fact against what's already known
  • Cheap token-overlap heuristic handles the clear cases; an optional small-model resolver is consulted only for the genuinely ambiguous band
  • Superseded facts are archived, not deleted -- visible as history, excluded from active recall
  • Retrieval today is lexical plus time-decay, not embedding-based -- semantic recall is explicitly not yet built

Memory model

  • A write is classified before it's stored: add (genuinely new), update (supersedes an existing fact), or duplicate (already known)
  • The heuristic is free and handles most cases; the model-backed resolver is the expensive path, used only when token overlap can't decide
  • The resolver's system prompt is cached and it uses a small, fast model -- the ambiguous band is the only place cost is spent
  • Any resolver failure falls back to the heuristic rather than blocking the write -- availability over completeness
  • A background job prunes archived facts past a configurable retention window, independent of the read-time decay used in ranking

Implementation

  • Single Go binary, SQLite-backed -- no external services, no Python runtime, one process you own
  • REST API: write, search, explicit supersede, hard delete, and a GDPR-style per-user erasure endpoint
  • conflict-lens factored out as its own standalone, dependency-free module -- reusable outside MemKit, one tested implementation rather than two
  • memkit-mcp: a dependency-free stdio MCP bridge, so any MCP client can use MemKit as long-term memory without talking REST directly
  • Multi-tenant via a per-tenant identifier; all data additionally scoped by user id

Evaluation methodology

M2 has run (see results below), against a real MemKit instance and real baseline implementations, not simulated. M1, M3, and M4 are still the planned design, kept here so the plan stays checkable against what eventually gets published, not retrofitted after the fact.

Workloads
  • Multi-week operational assistant: recurring entities, facts that change over time
  • Long incident thread: high-volume, low-signal history with a few decisive facts
  • Contradiction workload: facts deliberately updated and reversed across sessions
Baselines
  • Naive conversation-history retrieval
  • Vector-only retrieval (embeddings + top-k)
Metrics
  • Retrieval precision and recall against labelled gold memories
  • Stale-retrieval rate and contradiction rate per 100 queries
  • Context tokens consumed per task
  • Storage growth per 1k interactions
  • Retrieval and write-path latency (p50 / p95)

Experiments

Planned M1 -- Retrieval quality

Labelled gold memories across three workloads (multi-week assistant, long incident thread, contradiction workload) queried under MemKit, naive history retrieval, and vector-only top-k.

Measures Precision, recall, and answer relevance under a fixed context budget. Not yet built -- no gold labels exist yet.

M2 -- Staleness and contradiction

Facts deliberately updated and reversed across sessions; queries ask for the current value.

Measures Stale-retrieval rate and contradiction rate per 100 queries per approach. Run: 400-query synthetic contradiction workload, MemKit vs raw-history-replay vs vector-only. See results below.

Planned M3 -- Growth and cost

Identical interaction streams replayed to each approach over a simulated multi-week window.

Measures Storage growth, context tokens consumed per task, write-path latency and cost. Not yet run.

Planned M4 -- Resolver ablation

MemKit with the model-backed resolver disabled, heuristic only, versus enabled.

Measures Which share of the lexical-blindness gap the resolver actually closes, and at what added latency. Not yet run.

Results

Preliminary results
  • Write-time conflict resolution measurably cuts stale and contradictory retrieval: 52.0% stale-retrieval rate and 51.7% contradiction rate, vs 80.5% / 75.5% for a vector-similarity baseline and 100% / 100% for raw history replay -- 400-query synthetic contradiction workload, heuristic resolver only, no model-backed resolver yet
  • That gap isn't from returning less: recall stayed comparable across all three (0.865 MemKit vs 0.810 vector-only vs 1.000 raw history, which trivially recalls everything including every stale fact)
  • Real cost, not hidden: MemKit's write-path update cost measured 575ms median on real operational data (n=11), vs near-zero for both baselines -- the actual price of write-time classification
  • A fourth baseline (mem0, self-hosted/local mode) is measurably slower on both axes that ran: 90ms median query latency and roughly 4s median write-path update cost on the same real operational data, both well above MemKit's. Its stale/contradiction-rate numbers aren't available yet -- that run hit a rare internal error partway through a 1200-event synthetic pass and is being re-run, not silently omitted

Synthetic contradiction workload (400 deterministic query pairs, fictional entities) plus a small real-data set from the author's own operational logs, meeting this project's own minimum-cell-size floor for real data. The model-backed resolver's own contribution is still running, and mem0's stale/contradiction-rate comparison is incomplete for the reason noted above -- marked preliminary until both land.

Failure cases

  • No formal failure-case catalogue exists yet -- the evaluation below hasn't run, so this list would be invented rather than observed
  • The known, disclosed limitation in the meantime: the heuristic alone cannot see a lexically-dissimilar contradiction unless the resolver is enabled and the case falls in its ambiguous band

Limitations

  • No semantic or embedding-based recall yet -- retrieval today is lexical plus time-decay, a real and current gap, not a hypothetical one
  • SQLite only; no shared/concurrent-writer backend yet, which bounds this to single-agent or low-concurrency use for now
  • Conflict resolution assumes a single authoritative writer per fact domain -- multi-writer conflict has no defined resolution yet
  • The numbers above are heuristic-resolver-only -- how much of the gap the model-backed resolver closes on top of that hasn't been measured yet

Current status

v0.1: REST API, SQLite storage, conflict-lens (heuristic plus optional model-backed resolver for ambiguous cases) shipped and in use. Consolidation pruning and the MCP bridge are both live. First formal evaluation (staleness and contradiction, heuristic resolver) has run -- see results below; the model-backed resolver's own contribution is still being measured.

Roadmap

  • Postgres backend for shared/concurrent use
  • gRPC interface
  • Embedding-backed recall

Future research

  • Whether structured conflict resolution generalizes past single-writer, single-tenant workloads
  • Multi-agent shared memory with conflict resolution across writers
  • Memory as a causal model rather than a retrieval problem