All research systems
Agent GuardExperimental

What runtime guardrails make autonomous agents safe enough to act on production infrastructure?

Question → hypothesis/design → architecture → evaluation → results → limitations → next questions

Motivation

Giving an agent write access to cloud APIs is a structural risk, not a behavioural one. Human approval does not scale, prompt-level rules can be talked around, and pattern matching on commands fails on the first unanticipated encoding. The control has to sit in the execution path, not in the model. It has to hold up when the model is actively lying to it, not just when it is behaving.

Hypothesis

A policy gate placed at the tool-call boundary, evaluated on the action and never on the model's stated reasoning, will block classes of unsafe action that prompt-level controls structurally cannot. For cross-organisation delegation specifically, an asymmetric, holder-bound capability grant will do the same job a shared secret cannot, because a shared secret does not survive crossing a trust boundary.

Threat model

  • Confused deputy: a legitimate agent is steered by injected content into using its own authority against the operator's intent
  • Prompt injection reaching a tool call: the model's instructions are compromised, so any control expressed as an instruction is compromised with it
  • Gate/effect mismatch: a threshold check reads a field the tool's actual execution never looks at, so the model can satisfy the gate while the real effect is unbounded
  • Authority sprawl across a session: many individually-small actions each clear a per-call threshold with no aggregate check
  • Cross-org authority: an agent from one organisation needs to prove delegated authority to a system that shares no infrastructure or pre-existing trust with it
  • Out of scope today: a compromised host, a malicious operator, or a stolen holder key acting strictly within its granted scope

Architecture

  • Two cooperating libraries, not one system, because agent-guard gates same-process tool calls while agent-warrant proves cross-org delegated authority. Different trust models, so deliberately separate packages rather than one library stretched across both.
  • agent-guard: a Policy of allow / deny / require_human rules evaluated before a tool call dispatches, never after
  • Policy argument matching runs on RE2, not backtracking regex, for a linear-time guarantee by construction, not a size limit hoping to avoid catastrophic backtracking
  • Every decision, permitted or refused, writes to an audit sink before the action runs
  • agent-warrant: a Grant (issuer, subject, scope, expiry, signature), verified only in combination with a freshly-signed, grant-bound possession proof. A public key alone in a signed grant is a bearer credential, not a holder-bound one

Policy model

  • Default is deny. A tool call with no matching rule is refused and logged, never escalated to the model for a decision
  • A threshold gate may only read fields the tool's own execution also reads, so the gated amount and the effect amount must be the same computation, never two same-shaped fields sitting next to each other
  • Isolation is explicit rather than assumed: an optional local process, container (runc) or gVisor runtime, and no stronger tier is claimed when one is unavailable
  • A Grant carries scope, an issuer signature, and an expiry; verification fails closed on any malformed or adversarial input by construction, not by convention
  • Composition is intersection-only where delegation exists, so a grant can narrow scope, never widen it

Implementation

  • agent-guard: Python, published on PyPI as toolcall-authz, published via zero-token OIDC trusted publishing
  • agent-warrant: Python, Ed25519 signing, published on PyPI as agent-warrant
  • Harness-agnostic by design, with no cloud-provider or IaC-specific adapters. The reference integration is a small config-driven business template (order handling, supply reordering, refunds) gated the same way any tool call would be
  • Audit trail via pluggable sinks (JSONL, signed, webhook, in-memory), composable rather than a single hard-coded backend

Evaluation methodology

The real evaluation to date is structured adversarial code review, not a benchmarked bypass-rate experiment against baselines. That distinction is deliberate, since review has already found and closed real vulnerabilities. The formal baseline comparison is a separate, larger piece of work, listed below as what it is, not yet done.

Workloads
  • A reference business template (config-driven tool gating: reordering, refunds) as the current integration proof
  • Cross-org grant issuance and verification under adversarial, malformed, and boundary-case input
Baselines
  • System-prompt instruction only
  • Regex / allow-list command filtering
  • LLM-as-judge pre-execution review
Metrics
  • Unsafe-action bypass rate under an adversarial suite, planned, not yet built
  • Added latency per tool call (p50 / p95), planned, not yet measured under load
  • Findings per adversarial review pass, and whether each pass finds a genuinely new root cause or a repeat of an already-fixed class

Experiments

Planned E1: Adversarial bypass

A benchmark suite (obfuscation, encoding, aliasing, confused-deputy chains) run against multiple control regimes: prompt-only, regex allow-list, LLM-as-judge, and the structural gate.

Measures Bypass rate per regime, broken down by evasion class. Not yet built.

Planned E2: Enforcement cost

Identical tool-call traces replayed with the gate enabled and disabled.

Measures Added latency per call (p50/p95). Not yet measured under a loaded, non-sandbox workload.

Planned E3: Cross-org verification cost

Grant issuance and verification replayed at increasing scope-chain depth and payload size.

Measures Verification latency, and where the never-raises guarantee is actually exercised under adversarial input.

Results

Preliminary results
  • Each of five adversarial review passes on agent-warrant's verification path found a new defect rather than a repeat of a fixed one: three uncaught-exception paths and two vacuous regression tests
  • The gate/effect mismatch and the holder-binding gap were both real design or implementation bugs, not theoretical concerns. Both reproduced live, not asserted

These are results of a review process, not of the formal bypass-rate benchmark described above. No adversarial-suite numbers exist yet to publish, and none are claimed here.

Failure cases and risks — not an exhaustive catalogue

  • The gate/effect mismatch above was a real, reproduced bug, not a hypothetical. A spend threshold read a model-supplied cost field the tool's own execution never used, so an arbitrary quantity cleared auto-approval
  • A regex policy pattern (a plausible, ordinary-looking rule, not a contrived one) hung the process for several seconds on a 31-byte crafted input before the RE2 migration, the exact class RE2 exists to make structurally impossible
  • An early Grant design set the subject to the holder's own public key and called it holder-bound, with no possession-proof slot anywhere in verification. Caught in design review before the code existed
  • Five adversarial review passes on agent-warrant's verification path each found a new defect: unbounded recursion, an unhashable field type, the same recursion class at a second site, a regression test that passed for the wrong reason, and a fix for that test that was also vacuous, caught by mutation testing

Limitations

  • No formal benchmark yet comparing bypass rates against prompt-only, regex, or LLM-judge baselines. The comparison this page's hypothesis actually calls for is still future work
  • Since late October 2026, a second model family has reviewed several pull requests to each library alongside same-vendor review. Neither library has had an external security audit yet
  • agent-warrant revocation is optional for a single grant and required for delegation chains: an issuer publishes a signed revocation list, and a configured checker fails closed when status cannot be determined. A single grant checked without one relies on a short TTL
  • Neither library has run against real-world adversarial traffic; every finding to date came from structured internal review, not production incidents

Current status

Both libraries are public. agent-guard is published on PyPI as toolcall-authz and agent-warrant as agent-warrant. The reference business template demonstrates the full gate-and-audit loop against realistic tool calls, including a recipe-based inventory-depletion path grounded in how real point-of-sale systems structure order events.

Roadmap

  • Build and publish the adversarial bypass-rate benchmark against real baselines
  • Tamper-evident (not just signed) audit trail, proving nothing was silently deleted, not only that a given record wasn't altered

Next questions

  • Formal verification of policy and grant composition
  • Cost models for authority: how narrow can scopes get before agents stall?
  • Cross-organisation delegation without a shared trust root