All research systems
Agent GuardExperimental

What runtime guardrails make autonomous agents safe enough to act on production infrastructure?

Question → hypothesis/design → architecture → evaluation → results → limitations → next questions

Motivation

Giving an agent write access to cloud APIs is a structural risk, not a behavioural one. Human approval does not scale, prompt-level rules can be talked around, and pattern matching on commands fails on the first unanticipated encoding. The control has to sit in the execution path, not in the model. It has to hold up when the model is actively lying to it, not just when it is behaving.

Hypothesis

A policy gate placed at the tool-call boundary, evaluated on the action and never on the model's stated reasoning, will block classes of unsafe action that prompt-level controls structurally cannot. For cross-organisation delegation specifically, an asymmetric, holder-bound capability grant will do the same job a shared secret cannot, because a shared secret does not survive crossing a trust boundary.

Threat model

  • Confused deputy: a legitimate agent is steered by injected content into using its own authority against the operator's intent
  • Prompt injection reaching a tool call: the model's instructions are compromised, so any control expressed as an instruction is compromised with it
  • Gate/effect mismatch: a threshold check reads a field the tool's actual execution never looks at, so the model can satisfy the gate while the real effect is unbounded
  • Authority sprawl across a session: many individually-small actions each clear a per-call threshold with no aggregate check
  • Cross-org authority: an agent from one organisation needs to prove delegated authority to a system that shares no infrastructure or pre-existing trust with it
  • Out of scope today: a compromised host, a malicious operator, or a stolen holder key acting strictly within its granted scope

Architecture

  • Two cooperating libraries, not one system, because agent-guard gates same-process tool calls while agent-warrant proves cross-org delegated authority. Different trust models, so deliberately separate packages rather than one library stretched across both.
  • agent-guard: a Policy of allow / deny / require_human rules evaluated before a tool call dispatches, never after
  • Policy argument matching runs on RE2, not backtracking regex, for a linear-time guarantee by construction, not a size limit hoping to avoid catastrophic backtracking
  • Every decision, permitted or refused, writes to an audit sink before the action runs
  • agent-warrant: a Grant (issuer, subject, scope, expiry, signature), verified only in combination with a freshly-signed, grant-bound possession proof. A public key alone in a signed grant is a bearer credential, not a holder-bound one

Policy model

  • Default is deny. A tool call with no matching rule is refused and logged, never escalated to the model for a decision
  • A threshold gate may only read fields the tool's own execution also reads, so the gated amount and the effect amount must be the same computation, never two same-shaped fields sitting next to each other
  • Trust tiers exist (process-local today; container and micro-VM tiers defined) so the isolation boundary is explicit rather than assumed
  • A Grant carries scope, an issuer signature, and an expiry; verification fails closed on any malformed or adversarial input by construction, not by convention
  • Composition is intersection-only where delegation exists, so a grant can narrow scope, never widen it

Implementation

  • agent-guard: Python, published on PyPI as toolcall-authz, published via zero-token OIDC trusted publishing
  • agent-warrant: Python, Ed25519 signing, public on GitHub, not yet published to PyPI
  • Harness-agnostic by design, with no cloud-provider or IaC-specific adapters. The reference integration is a small config-driven business template (order handling, supply reordering, refunds) gated the same way any tool call would be
  • Audit trail via pluggable sinks (JSONL, signed, webhook, in-memory), composable rather than a single hard-coded backend

Evaluation methodology

The real evaluation to date is structured adversarial code review, not a benchmarked bypass-rate experiment against baselines. That distinction is deliberate, since review has already found and closed real vulnerabilities. The formal baseline comparison is a separate, larger piece of work, listed below as what it is, not yet done.

Workloads
  • A reference business template (config-driven tool gating: reordering, refunds) as the current integration proof
  • Cross-org grant issuance and verification under adversarial, malformed, and boundary-case input
Baselines
  • System-prompt instruction only
  • Regex / allow-list command filtering
  • LLM-as-judge pre-execution review
Metrics
  • Unsafe-action bypass rate under an adversarial suite, planned, not yet built
  • Added latency per tool call (p50 / p95), planned, not yet measured under load
  • Findings per adversarial review pass, and whether each pass finds a genuinely new root cause or a repeat of an already-fixed class

Experiments

Planned E1: Adversarial bypass

A benchmark suite (obfuscation, encoding, aliasing, confused-deputy chains) run against multiple control regimes: prompt-only, regex allow-list, LLM-as-judge, and the structural gate.

Measures Bypass rate per regime, broken down by evasion class. Not yet built.

Planned E2: Enforcement cost

Identical tool-call traces replayed with the gate enabled and disabled.

Measures Added latency per call (p50/p95). Not yet measured under a loaded, non-sandbox workload.

Planned E3: Cross-org verification cost

Grant issuance and verification replayed at increasing scope-chain depth and payload size.

Measures Verification latency, and where the never-raises guarantee is actually exercised under adversarial input.

Results

Preliminary results
  • Every adversarial review pass on agent-warrant's verification path to date has found a genuinely distinct root cause, not a repeat of the last one. Four passes, four different uncaught-exception classes, each closed and mutation-tested before the next pass
  • The gate/effect mismatch and the holder-binding gap were both real design or implementation bugs, not theoretical concerns. Both reproduced live, not asserted

These are results of a review process, not of the formal bypass-rate benchmark described above. No adversarial-suite numbers exist yet to publish, and none are claimed here.

Failure cases and risks — not an exhaustive catalogue

  • The gate/effect mismatch above was a real, reproduced bug, not a hypothetical. A spend threshold read a model-supplied cost field the tool's own execution never used, so an arbitrary quantity cleared auto-approval
  • A regex policy pattern (a plausible, ordinary-looking rule, not a contrived one) hung the process for several seconds on a 31-byte crafted input before the RE2 migration, the exact class RE2 exists to make structurally impossible
  • An early Grant design set the subject to the holder's own public key and called it holder-bound, with no possession-proof slot anywhere in verification. Caught in design review before the code existed
  • Iterative adversarial review of agent-warrant's verification path found a genuinely different uncaught-exception class on four separate passes: unbounded recursion, an unhashable field type, a second independent recursion path, and a regression test that passed for the wrong reason

Limitations

  • No formal benchmark yet comparing bypass rates against prompt-only, regex, or LLM-judge baselines. The comparison this page's hypothesis actually calls for is still future work
  • Every review pass so far has been conducted by the same reviewing lineage; no independent cross-vendor review has run yet
  • agent-warrant has no revocation mechanism before a grant's own expiry. A short TTL is the only containment today, a stated design choice, not an oversight
  • Neither library has run against real-world adversarial traffic; every finding to date came from structured internal review, not production incidents

Current status

Both libraries are public. agent-guard is published on PyPI; agent-warrant is public on GitHub. The reference business template demonstrates the full gate-and-audit loop against realistic tool calls, including a recipe-based inventory-depletion path grounded in how real point-of-sale systems structure order events.

Roadmap

  • Build and publish the adversarial bypass-rate benchmark against real baselines
  • Tamper-evident (not just signed) audit trail, proving nothing was silently deleted, not only that a given record wasn't altered
  • Multi-hop delegation chains for agent-warrant, once a real second consumer needs it

Next questions

  • Formal verification of policy and grant composition
  • Cost models for authority: how narrow can scopes get before agents stall?
  • Cross-organisation delegation without a shared trust root