Research Journal

Notes from the workbench.

Short, dated entries on what I'm learning, what's breaking, and what questions are sticking around.

2026-08-01

August 2026 — Building memory isn't enough

The harder problem is deciding what should be remembered, and what should be allowed to fade.

We have spent the last month implementing write-time conflict classification and storage for MemKit -- not embedding-based retrieval yet, that's still ahead. The code works. The harder problem is deciding what should be remembered, and what should be allowed to fade.

Human memory is reconstructive and lossy by design. We do not store raw experience; we store summaries, emotions, and predictions. Machine memory should probably work the same way, but we do not yet know the right compression function for agent experience.

I am starting to think that memory is really a planning problem. You remember what you expect to need. The rest is noise. If that is true, then memory systems and goal systems cannot be designed independently.

2026-07-05

July 2026 — Uncertainty is information, not a bug

The most interesting failures are not when an agent is wrong, but when it is uncertain and acts anyway.

Agent Guard is running as a policy gate in front of a small reference business template -- not yet an infrastructure impact estimator, that's a harder, later problem. The most interesting failures so far have not been the model doing something wrong; they've been the review process finding a gate that checked the wrong field, or a credential design that looked holder-bound but wasn't.

Uncertainty is not a bug; it is information. The question is what to do with it. We are testing a model where uncertainty escalates to a human, but with a structured proposal rather than a raw alert: intended action, predicted impact, and the agent's confidence.

This feels like the right granularity. Agents should not ask permission for every step. They should ask when the expected cost of being wrong exceeds the cost of waiting.

2026-06-12

June 2026 — Designing the benchmark before running it

No formal benchmark has run yet on Vigilo. Writing down what it measures before the first result exists is the point.

No formal benchmark has run yet on Vigilo -- that's still ahead. What is built and running: a daemon watching file, process, and network signals with a one-second alert path, and a correlating pass every five minutes. The open question worth designing an actual experiment around: can the five-minute correlation catch attack chains that per-event rules structurally cannot, without drowning the operator in false positives.

The metric that actually matters is not detection rate in isolation -- a rule that fires on everything has perfect recall and is useless. It's the fraction of scripted multi-step chains caught only by correlation, against the false-positive rate the operator has to live with on ordinary production traffic over weeks, not a single clean test run.

The bigger lesson is that we need evaluation environments where agents can fail safely, and a harness that's designed before the first run rather than fitted to whatever result comes back. Production is too expensive and too slow to iterate in, so the harness is part of the research, not overhead around it.