Most agent 'memory' is vector search over chat history. That is context management, not memory. 'I love my job' and 'I quit' both mention the job and retrieve together on similarity alone -- a naive agent synthesizes both into an answer that's wrong the moment it's assembled. Similarity is not truth. MemKit's starting bet is that solving that problem comes before optimizing recall quality, not after -- the current build deliberately has no semantic search yet, because a fast wrong answer is worse than a slightly slower right one.
Explicit, write-time conflict classification -- resolving whether a new fact adds, updates, or duplicates what's already known -- will remove stale and contradictory retrievals that naive history replay and similarity-only retrieval structurally cannot catch, at a write-path cost worth paying.
M2 has run (see results below), against a real MemKit instance and real baseline implementations, not simulated. M1, M3, and M4 are still the planned design, kept here so the plan stays checkable against what eventually gets published, not retrofitted after the fact.
Labelled gold memories across three workloads (multi-week assistant, long incident thread, contradiction workload) queried under MemKit, naive history retrieval, and vector-only top-k.
Measures Precision, recall, and answer relevance under a fixed context budget. Not yet built -- no gold labels exist yet.
Facts deliberately updated and reversed across sessions; queries ask for the current value.
Measures Stale-retrieval rate and contradiction rate per 100 queries per approach. Run: 400-query synthetic contradiction workload, MemKit vs raw-history-replay vs vector-only. See results below.
Identical interaction streams replayed to each approach over a simulated multi-week window.
Measures Storage growth, context tokens consumed per task, write-path latency and cost. Not yet run.
MemKit with the model-backed resolver disabled, heuristic only, versus enabled.
Measures Which share of the lexical-blindness gap the resolver actually closes, and at what added latency. Not yet run.
Synthetic contradiction workload (400 deterministic query pairs, fictional entities) plus a small real-data set from the author's own operational logs, meeting this project's own minimum-cell-size floor for real data. The model-backed resolver's own contribution is still running, and mem0's stale/contradiction-rate comparison is incomplete for the reason noted above -- marked preliminary until both land.
v0.1: REST API, SQLite storage, conflict-lens (heuristic plus optional model-backed resolver for ambiguous cases) shipped and in use. Consolidation pruning and the MCP bridge are both live. First formal evaluation (staleness and contradiction, heuristic resolver) has run -- see results below; the model-backed resolver's own contribution is still being measured.