All research systems
Agent GuardExperimental

What runtime guardrails make autonomous agents safe enough to act on production infrastructure?

Motivation

Giving an agent write access to cloud APIs is a structural risk, not a behavioural one. Human approval does not scale, prompt-level rules can be talked around, and pattern matching on commands fails on the first unanticipated encoding. The control has to sit in the execution path, not in the model — and it has to hold up when the model is actively lying to it, not just when it is behaving.

Hypothesis

A policy gate placed at the tool-call boundary — evaluated on the action, never on the model's stated reasoning — will block classes of unsafe action that prompt-level controls structurally cannot. For cross-organisation delegation specifically, an asymmetric, holder-bound capability grant will do the same job a shared secret cannot, because a shared secret does not survive crossing a trust boundary.

Threat model

  • Confused deputy: a legitimate agent is steered by injected content into using its own authority against the operator's intent
  • Prompt injection reaching a tool call: the model's instructions are compromised, so any control expressed as an instruction is compromised with it
  • Gate/effect mismatch: a threshold check reads a field the tool's actual execution never looks at, so the model can satisfy the gate while the real effect is unbounded
  • Authority sprawl across a session: many individually-small actions each clear a per-call threshold with no aggregate check
  • Cross-org authority: an agent from one organisation needs to prove delegated authority to a system that shares no infrastructure or pre-existing trust with it
  • Out of scope today: a compromised host, a malicious operator, or a stolen holder key acting strictly within its granted scope

Architecture

  • Two cooperating libraries, not one system: agent-guard gates same-process tool calls; agent-warrant proves cross-org delegated authority. Different trust models, so deliberately separate packages rather than one library stretched across both.
  • agent-guard: a Policy of allow / deny / require_human rules evaluated before a tool call dispatches, never after
  • Policy argument matching runs on RE2, not backtracking regex — a linear-time guarantee by construction, not a size limit hoping to avoid catastrophic backtracking
  • Every decision — permitted or refused — writes to an audit sink before the action runs
  • agent-warrant: a Grant (issuer, subject, scope, expiry, signature), verified only in combination with a freshly-signed, grant-bound possession proof — a public key alone in a signed grant is a bearer credential, not a holder-bound one

Policy model

  • Default is deny: a tool call with no matching rule is refused and logged, never escalated to the model for a decision
  • A threshold gate may only read fields the tool's own execution also reads — the gated amount and the effect amount must be the same computation, never two same-shaped fields sitting next to each other
  • Trust tiers exist (process-local today; container and micro-VM tiers defined) so the isolation boundary is explicit rather than assumed
  • A Grant carries scope, an issuer signature, and an expiry; verification fails closed on any malformed or adversarial input by construction, not by convention
  • Composition is intersection-only where delegation exists: a grant can narrow scope, never widen it

Implementation

  • agent-guard: Python, published on PyPI as toolcall-authz, published via zero-token OIDC trusted publishing
  • agent-warrant: Python, Ed25519 signing, public on GitHub, not yet published to PyPI
  • Harness-agnostic by design: no cloud-provider or IaC-specific adapters. The reference integration is a small config-driven business template (order handling, supply reordering, refunds) gated the same way any tool call would be
  • Audit trail via pluggable sinks (JSONL, signed, webhook, in-memory), composable rather than a single hard-coded backend

Evaluation methodology

The real evaluation to date is structured adversarial code review, not a benchmarked bypass-rate experiment against baselines. That distinction is deliberate: review has already found and closed real vulnerabilities; the formal baseline comparison is a separate, larger piece of work, listed below as what it is — not yet done.

Workloads
  • A reference business template (config-driven tool gating: reordering, refunds) as the current integration proof
  • Cross-org grant issuance and verification under adversarial, malformed, and boundary-case input
Baselines
  • System-prompt instruction only
  • Regex / allow-list command filtering
  • LLM-as-judge pre-execution review
Metrics
  • Unsafe-action bypass rate under an adversarial suite — planned, not yet built
  • Added latency per tool call (p50 / p95) — planned, not yet measured under load
  • Findings per adversarial review pass, and whether each pass finds a genuinely new root cause or a repeat of an already-fixed class

Experiments

Planned E1 — Adversarial bypass

A benchmark suite (obfuscation, encoding, aliasing, confused-deputy chains) run against multiple control regimes: prompt-only, regex allow-list, LLM-as-judge, and the structural gate.

Measures Bypass rate per regime, broken down by evasion class. Not yet built.

Planned E2 — Enforcement cost

Identical tool-call traces replayed with the gate enabled and disabled.

Measures Added latency per call (p50/p95). Not yet measured under a loaded, non-sandbox workload.

Planned E3 — Cross-org verification cost

Grant issuance and verification replayed at increasing scope-chain depth and payload size.

Measures Verification latency, and where the never-raises guarantee is actually exercised under adversarial input.

Results

Preliminary results
  • Every adversarial review pass on agent-warrant's verification path to date has found a genuinely distinct root cause, not a repeat of the last one — four passes, four different uncaught-exception classes, each closed and mutation-tested before the next pass
  • The gate/effect mismatch and the holder-binding gap were both real design or implementation bugs, not theoretical concerns — both reproduced live, not asserted

These are results of a review process, not of the formal bypass-rate benchmark described above. No adversarial-suite numbers exist yet to publish, and none are claimed here.

Failure cases

  • The gate/effect mismatch above was a real, reproduced bug, not a hypothetical: a spend threshold read a model-supplied cost field the tool's own execution never used, so an arbitrary quantity cleared auto-approval
  • A regex policy pattern (a plausible, ordinary-looking rule, not a contrived one) hung the process for several seconds on a 31-byte crafted input before the RE2 migration — the exact class RE2 exists to make structurally impossible
  • An early Grant design set the subject to the holder's own public key and called it holder-bound with no possession-proof slot anywhere in verification — caught in design review before the code existed
  • Iterative adversarial review of agent-warrant's verification path found a genuinely different uncaught-exception class on four separate passes — unbounded recursion, an unhashable field type, a second independent recursion path, and a regression test that passed for the wrong reason

Limitations

  • No formal benchmark yet comparing bypass rates against prompt-only, regex, or LLM-judge baselines — the comparison this page's hypothesis actually calls for is still future work
  • Every review pass so far has been conducted by the same reviewing lineage; no independent cross-vendor review has run yet
  • agent-warrant has no revocation mechanism before a grant's own expiry — a short TTL is the only containment today, a stated design choice, not an oversight
  • Neither library has run against real-world adversarial traffic; every finding to date came from structured internal review, not production incidents

Current status

Both libraries are public. agent-guard is published on PyPI; agent-warrant is public on GitHub. The reference business template demonstrates the full gate-and-audit loop against realistic tool calls, including a recipe-based inventory-depletion path grounded in how real point-of-sale systems structure order events.

Roadmap

  • Build and publish the adversarial bypass-rate benchmark against real baselines
  • Tamper-evident (not just signed) audit trail — proving nothing was silently deleted, not only that a given record wasn't altered
  • Multi-hop delegation chains for agent-warrant, once a real second consumer needs it

Future research

  • Formal verification of policy and grant composition
  • Cost models for authority: how narrow can scopes get before agents stall?
  • Cross-organisation delegation without a shared trust root