Question → hypothesis/design → architecture → evaluation → results → limitations → next questions
Most agent 'memory' is vector search over chat history. That is context management, not memory. 'I love my job' and 'I quit' both mention the job and retrieve together on similarity alone, so a naive agent synthesizes both into an answer that's wrong the moment it's assembled. Similarity is not truth. MemKit's starting bet is that solving that problem comes before optimizing recall quality, not after. The current build deliberately has no semantic search yet, because a fast wrong answer is worse than a slightly slower right one.
Explicit, write-time conflict classification, resolving whether a new fact adds, updates, or duplicates what's already known, will remove stale and contradictory retrievals that naive history replay and similarity-only retrieval structurally cannot catch, at a write-path cost worth paying.
M2 has run on a held-out workload (see results below), against a real MemKit instance and real baseline implementations, after an earlier published result failed to reproduce. M4 has partly run. M1 and M3 are still the planned design, kept here so the plan stays checkable against what eventually gets published, not retrofitted after the fact.
Labelled gold memories across three workloads (multi-week assistant, long incident thread, contradiction workload) queried under MemKit, naive history retrieval, and vector-only top-k.
Measures Precision, recall, and answer relevance under a fixed context budget. Not yet built. No gold labels exist yet.
Facts updated, reworded, restated, shared between different people, and added alongside other attributes; queries ask for the current value or for a fact that should have survived.
Measures Stale rate, and the rate at which a fact that should survive is missing from the top five results, per category and approach. Run: a held-out synthetic workload (development and test splits, 102 queries per category), heuristic only vs a local-model resolver vs vector retrieval. See results below.
Identical interaction streams replayed to each approach over a simulated multi-week window.
Measures Storage growth, context tokens consumed per task, write-path latency and cost. Not yet run.
Heuristic only versus a local Ollama judge (llama3.1:8b) on the held-out workload. The Claude-based resolver was not evaluated.
Measures Reported under results: lost facts fell sharply, stale answers did not, at a median 4.6 s per write.
Synthetic workloads only (fictional entities), one run per configuration, development iterated four times, and one text-only diagnostic was computed on the test split before settings were frozen. The local-model resolver and the evaluation harness are in an open pull request (github.com/agent-rails/memkit/pull/2), which holds the full protocol and disclosures. Marked preliminary.
v0.1: REST API, SQLite storage, conflict-lens (heuristic plus optional model-backed resolver for ambiguous cases) shipped and in use. Consolidation pruning and the MCP bridge are both live. A held-out evaluation has run and corrected an earlier published result (see results below). A search fix has merged; a local-model resolver and the evaluation harness are in an open pull request.