Kill Switches for Autonomous Agents
Spend caps, blast-radius control, and the security model for an agent that holds your keys
13 min read
Spend caps, blast-radius control, and the security model for an agent that holds your keys
"Designing for the misbehaving cycle, not the well-behaved one"
Illustration by Nano Banana 2 (fal.ai)
In Part 2 I described the agent I run unattended: a daemon that wakes every 15 minutes, reads the state of my projects, does a bounded slice of work, and writes down what happened. That post spent one section on guardrails. This is that section, expanded into what I’d hand someone before they turned their own agent loose.
An autonomous agent is not a smarter chatbot. It’s a process that holds three dangerous capabilities at once: a paid model API key (it can spend), a shell (it can execute code), and outbound channels that speak as me (it can impersonate and make noise). Any one of those, unsupervised, is a liability. All three in one always-on process is a different category of risk.
“Can it be jailbroken” is the question for a model you serve to strangers. For an agent you run yourself, the question is what the blast radius is when it misbehaves, because software running thousands of times a day on real inputs eventually hits the one you didn’t plan for. Design for the misbehaving cycle, not the well-behaved one.
This post is the controls that bound those three capabilities. None are exotic. All are load-bearing.
Three capabilities, three failure shapes.
The key: it can spend. The agent holds a credential for a paid model API. A retry loop, a runaway plan, or a quiet cache regression turns “a few cents a cycle” into a bill that climbs while no one is watching. There’s no error, no crash, and no alert by default. Only a meter running.
The shell: it can execute. The Act stage runs commands. A shell is the most general-purpose capability there is, which is why it’s the most dangerous to hand to a process acting on its own judgment. The damage is to your filesystem, your repos, and your machine, and it can be irreversible in a way a bad message never is.
The channels: it can speak as me. The agent sends messages, opens PRs, and posts updates under my identity. Misbehavior here is reputational: noise to real people, a wrong message with my name on it, or (the failure I’ll get to below) a loop that spams.
Feel free to send us a message with your thoughts, or learn more about us!
One principle sits under all three: build the OFF switch before you build the capability. If you can’t say how you’d stop a capability instantly, you’re not ready to give it to an unattended process.
A kill switch is only a kill switch if flipping it stops the thing. The common failure is a flag checked at startup, or behind a deploy, or in one process but not the others. That’s a config flag with good intentions. When an agent misbehaves you need it stopped now, not at the next restart.
The contract I hold:
Build order matters. For any new outbound capability, the off switch ships first and is tested before the agent can send on that channel. You will need it, usually at an inconvenient hour.
An unattended agent holding a paid credential needs a hard ceiling on spend, and the detail that matters more than the ceiling is what happens when the system is uncertain.
The budget kill switch trips when a ceiling is breached. That’s table stakes. The non-negotiable property is that it fails closed: if the system cannot confirm you’re under budget, it stops spending. Uncertainty resolves to “stop,” not “probably fine.”
The intuitive default is fail-open: if the budget check errors out, keep working so the agent stays useful. That’s backwards for an unattended spend path. A fail-open cap protects you only when everything is healthy, which is when you didn’t need it. The cap has to hold hardest when the system is confused, because a confused always-on agent with a live credential is what empties an account.
A spend check often leans on an estimate: a ratio that maps raw usage to dollars, or a recent measurement of what cycles cost. Trusted blindly, that input is a liability of its own. The rule: any estimate feeding the budget check must be freshness-gated. Trust it only if two things hold:
When the input fails either gate, fall back to raw numbers and let the kill switch act on those alone. Stale “we’re fine” data is worse than no data. No data makes you cautious. Stale reassurance makes you confident on bad information, and green-lights spend the current reality wouldn’t justify.
What stops the budget switch from being silently cleared by a bug, a race, or a well-meaning process that “resets” state? A separate watchdog guards the kill switch. If it sees the pause flag cleared without authorization, it reverts it. The control protecting your spend gets its own control. You don’t want to learn your budget cap was switched off from the invoice.
The shell is the broadest capability and earns the most paranoia. The Act stage, where the agent executes its plan, runs under limits enforced by the system rather than by the model’s good judgment, because the model’s judgment is the thing under test.
Allowlist-only shell. The agent can run a fixed set of binaries and nothing else. Not “block the dangerous ones” (you’ll never enumerate them all) but “permit the safe ones.” On top of that: no shell metacharacters, so a command can’t smuggle a second command past the allowlist, and a hard rule against piping into rm, sudo, or eval. The whole class of “the model composed a clever destructive one-liner” is closed structurally.
Restricted write paths. File writes are confined to two explicitly blessed directories: the agent’s scratch space and its own knowledge store. It cannot edit source it isn’t supposed to touch, cannot edit its own guardrails, and cannot reach the rest of the machine. A bad write is bounded to directories you’ve decided you can afford to lose.
SSRF protection. Any URL the agent fetches is checked first: localhost and private IP ranges are blocked. This stops a crafted input, a poisoned document, or a manipulated plan from steering the agent into your internal services, your cloud metadata endpoint, or your private network. An agent that can fetch arbitrary URLs is one prompt away from being a confused deputy inside your perimeter.
Per-process secret resolution. Secrets resolve at the moment of execution, in the process that needs them, and are never injected session-wide into the environment. Session-wide injection is convenient, but an ambient credential leaks into every log line, crash dump, child process, and error report. Resolved per process, a leaked environment is not a handed-over credential.
Denylists lose. To block dangerous behavior by enumeration you have to think of every dangerous thing in advance, and an agent generating commands from a language model will find the one you didn’t. Allowlists invert the burden: enumerate the small set of things the agent legitimately needs, and deny everything else by default. You’re far more likely to have a complete list of “what this agent does” than of “what could go wrong.”
The same shape shows up in every control in this post:
The instinct is default-deny, permit explicitly. For an unattended process acting on a model’s judgment, it’s the only posture that survives the input you didn’t anticipate.
The outbound channel has a failure mode the other two don’t: it can feed back into itself. The cleanest way to explain the rule is how I learned it.
The rule I now treat as non-negotiable: exactly one sender per channel, with cross-process deduplication in shared state. Both halves are required.
One sender per channel. For each outbound channel, exactly one process may call the send API. Not “primary and backup,” not “whichever is up.” The moment two processes can send on a channel you’ve created the conditions for the echo loop, and “they probably won’t collide” is not a security property.
Cross-process deduplication in shared state. Before any send, the sender checks shared state: has this message already gone out? The dedup record lives where every process can see it (a shared store with a short time-to-live), not in a per-process ring buffer. This is the half people skip.
A per-process dedup ring only knows what that process has sent. It is structurally blind to what a sibling did. The instant you have concurrent processes (multiple agents, multiple sessions, a failover that didn’t fully fail over), each one’s memory of “what I’ve sent” is an island, and the collisions happen in the water between them. You cannot solve a multi-process race with a single-process data structure.
This generalizes past messaging. Any contended resource (a channel, a queue, a credential, a lock) that multiple agents can touch needs one owner or shared coordination, ideally both. Every “the agents went haywire together” story I’ve heard reduces to a missing boundary like this one. Draw the boundary first.
The controls above bound what the agent can do. The trust ladder bounds what a new, unproven automation may do, so adding capabilities stays safe over time instead of becoming a series of held breaths.
New automations don’t act for real on day one. They run in shadow first: the automation observes, decides, proposes the exact action it would take, and logs the proposal without executing it. You get a running record of “here’s what I would have done” with none of the consequences.
It graduates on evidence. An automation moves from shadow to real after a threshold of correct proposals, measured, not vibes. “It seems to work” is the start of the evaluation, not the end.
This matters because an agent’s failure shape differs from a tool’s. An assistant’s mistake is one bad suggestion you reject. An agent’s mistake repeats every cycle until someone notices: confidently wrong, at scale, on a clock. Shadow mode is how you find out an automation is confidently wrong before it’s wired to consequences.
The last control catches the failure the others let through: a service that keeps dying and coming back. You want crashed services restarted. You don’t want a broken service flapping forever while an automatic restart loop papers over a real problem. The watchdog that keeps background services alive does two things that make it safe rather than persistent.
It caps restarts per hour. A service gets a small number of restarts in a window, three per hour in my setup. Past that, the watchdog stops trying, quarantines the service, and escalates to me. A service that has died three times in an hour isn’t unlucky. It’s broken, and “broken” needs a human, not an infinite loop of hopeful restarts. The cap turns resilience into a signal instead of a mask.
It respects operator intent. If I’ve deliberately disabled a service, the watchdog must not “helpfully” bring it back. An automation that fights a human’s explicit decision is worse than no automation, because now you’re wrestling your own infrastructure. So before any restart, the watchdog checks whether the service was intentionally disabled, and if so leaves it alone. The rule generalizes: automation should never override a deliberate human action. The operator’s explicit intent is the highest authority in the system.
Three dangerous capabilities, each bounded:
Two meta-controls keep the boundaries honest over time:
The thread through all of it: default-deny, permit explicitly, and build the OFF switch before the capability. Each control is the same move. Deny by default, permit the narrow known-good set, and make uncertainty resolve to “stop.”
One caveat. None of this makes an agent safe in the absolute sense. It makes the blast radius bounded and known, which is the more useful goal, because “safe” is a claim you can’t verify and “bounded” is a property you can design and test. I can say exactly how bad my worst case is: which directories a bad write can reach, what my spend ceiling is, how fast I can stop every send. That’s what lets me leave the agent running while I sleep. Not faith that it’ll behave. A known ceiling on what happens when it doesn’t.
The companion piece is cost. The spend cap is the hard backstop; what keeps the agent cheap before the cap ever trips is prompt-cache discipline, which gets the full treatment in Prompt Cache Invariants, the next post in this vein (September 15).
Map your agent's three capabilities: spend, execution, outbound. Draw one explicit boundary around each, and make every boundary fail toward stop. Build the off switch first, run new automations in shadow, and write down your worst case as a number. Then turn it on.
Running an agent that holds your keys, or about to? Tell me which capability scared you into building a control, and where the boundary turned out to be in the wrong place.