Prompt Cache Invariants: The Discipline That Keeps an AI Daemon Cheap
Why one cache-busting mutation can 10x a conversation — and the rules that stop it
15 min read
Why one cache-busting mutation can 10x a conversation — and the rules that stop it
"The prefix is the cache key — guard it"
Illustration by FLUX.2 Pro (fal.ai)
Feel free to send us a message with your thoughts, or learn more about us!
In Part 2 I made a claim and then walked past it: a single cache-busting mutation in a 30-turn conversation can 10x the cost of that conversation. This post is what that means.
The prompt cache is the difference between an autonomous agent you can afford to run constantly and one you can’t. At a $20/month budget, cache discipline isn’t an optimization you get to after the features land. It’s an invariant you design around from the first line, the same way you’d treat a memory-safety rule or a SQL-injection guard.
Every way you’d break it looks reasonable: injecting a timestamp, reformatting the system prompt, adding a tool mid-conversation, trimming old messages. Each one silently torches your cache-hit rate for the rest of the session.
This post covers the rules: why prefix caching keys the way it does, the four mutations that break it, the operations that are safe, and how to enforce it statically so violations fail loud.
Every compatible provider caches on the prefix of the request. Not the whole request: the prefix, evaluated in order:
┌─────────────────────────────┐
│ 1. System prompt │ ← cached prefix
│ 2. Tool schema (definitions)│ ← cached prefix
│ 3. Message history (turns) │ ← cached prefix
├─────────────────────────────┤
│ 4. The new user turn │ ← uncached (this is fine)
└─────────────────────────────┘The provider walks that prefix from the top and matches it, token by token, against what it cached on the previous request. As long as the bytes are identical up to some point, you get a cache hit for everything up to that point. You pay roughly 10% of the input cost for the cached span, and the response comes back about 3x faster because the model doesn’t re-process tokens it already has warm.
The catch is the phrase “up to some point.” The match is a prefix match, not a set match. The instant the bytes diverge (one changed character in the system prompt, one reordered tool, one edited word in an old message) the cache match stops at that point, and everything from there to the end of the request is a cache miss. Re-priced at full rate. Re-processed at full latency.
This is also why the cost blows up the way it does. A cache-busting mutation doesn’t make one turn expensive. It moves the divergence point earlier in the request, so the miss applies to the rest of that turn, and because the mutated prefix is now what gets cached, the same expensive prefix can keep missing on subsequent turns too if the non-determinism repeats. In a long conversation, you’re not paying a one-time penalty. You’re paying the penalty over and over, on a request that keeps growing.
A conversation’s input cost grows with each turn, because every turn resends the entire history as context. Turn 1 sends a small prefix. Turn 30 sends 30 turns’ worth. With caching working, almost all of that resent history is a cache hit. You pay ~10% on the warm span and full rate only on the new turn. That’s what makes long agent conversations affordable at all.
Now break the cache. Suppose something non-deterministic, say a timestamp in the system prompt, changes on every single turn. The system prompt sits at the very top of the prefix, so a change there moves the divergence point to byte zero. Every turn is now a full cache miss on the entire history. You’ve gone from paying ~10% on the growing context to paying 100% of it, every turn, for the life of the conversation.
That’s the shape of the 10x: the compounding cost of resending an ever-larger history at full price across a 30-turn conversation, when you could have been resending it at a tenth of the price.
On a $20/month self-evolution budget, the math isn’t academic. A cache-busting bug doesn’t make the agent “a bit pricier.” It makes the difference between an agent that runs every 15 minutes forever and one that trips its own budget kill switch by lunchtime. The cost ceiling is what lets the agent be autonomous in the first place, so anything that quietly multiplies cost is an attack on the autonomy itself.
Here’s the spine of the whole discipline. Four mutations, each one a way of reaching back into the frozen prefix. Each looks innocent. Each isn’t.
The system prompt is the very top of the prefix. Touch it and you’ve moved the divergence point to byte zero: the most expensive possible miss, because nothing after it can hit the cache. Once call_llm has been invoked once with a system prompt, the exact same string has to be reused on every subsequent turn of that conversation. Byte for byte.
The violations don’t look like “mutating the system prompt.” They look like helpfulness:
// ❌ LOOKS INNOCENT — concatenating "just a little" context
const system = BASE_SYSTEM_PROMPT + `\n\nCurrent focus: ${task}`;
// ^^^ different string every turn = byte-zero miss
// ❌ LOOKS INNOCENT — "harmless" reformatting between turns
const system = prettyPrintMarkdown(BASE_SYSTEM_PROMPT);
// ^^^ even whitespace changes are byte changes
// ❌ LOOKS INNOCENT — telling the model what time it is
const system = `${BASE_SYSTEM_PROMPT}\nIt is ${new Date().toISOString()
// ✅ CORRECT — one frozen string, captured at conversation start, reused verbatim
const system = BASE_SYSTEM_PROMPT; // identical bytes, every turn, no exceptionsThe fix is a discipline, not a clever trick: capture the system prompt once, freeze it, and never let anything append to it, reformat it, or inject into it mid-conversation. If the model needs to know the current task or the current time, that information goes in the last user turn, the one part of the request that’s supposed to change.
The tool schema sits right below the system prompt in the prefix. Adding a tool, removing a tool, or even reordering the tools between turns busts the cache from the tool-schema boundary onward. And it has a second cost the other mutations don’t: it confuses the model’s tool planning. A model that learned the tool set on turn 1 and finds a different one on turn 5 plans worse, not slower.
The trap here is the instinct to be “smart” about tools: hide the ones that aren’t relevant, surface new ones on demand:
// ❌ LOOKS INNOCENT — dynamically narrowing tools per turn to "help" the model
const tools = relevantToolsFor(currentTask); // different list each turn = cache miss + worse planning
// ❌ LOOKS INNOCENT — appending a tool because the model seemed to want it
tools.push(newlyDiscoveredTool); // mutates the schema mid-conversation// ✅ CORRECT — decide the full tool set ONCE, at conversation start, then freeze it
const tools = resolveToolSetForSession(); // computed once per session, never per turnThe rule: decide the tool set once per session, not per turn. If you need different tools, that’s a different conversation. This is exactly the cliffhanger I left at the end of Part 2 and the whole subject of the next post on MCP in production (November 15): your tool schema is part of the cache prefix, which means the common pattern of wiring tools in and out of an agent mid-session is quietly a cost bug. There’s a lot more to say there; it gets its own deep dive.
Every prior user and assistant turn is part of the frozen prefix, in order. Editing any earlier message, to compress it or redact it or “improve” it, moves the divergence point to that message and invalidates everything after it. The earlier the message you touch, the bigger the miss.
This one is seductive because it wears the mask of a good idea: context is getting long, so trim the old turns to save tokens.
// ❌ LOOKS INNOCENT — compressing old turns to save context window
history[2].content = summarize(history[2].content);
// ^^^ rewriting turn 2 invalidates turns 3..N on the very next request
// ❌ LOOKS INNOCENT — redacting something after the fact
history = history.map(redactSecrets); // every edited turn shifts the divergence point earlier// ✅ CORRECT — history is append-only; never reach back and edit
history.push(newUserTurn);
history.push(newAssistantTurn); // the prefix grows at the end, never in the middleYou don’t “improve” history. You append to it. If the conversation genuinely needs to get shorter, that is the compress operation I describe in the permitted section, not an edit, which ends the conversation rather than mutating it.
This is the umbrella that the timestamp example lives under, and it’s worth naming on its own because it shows up in places you don’t expect. Anything non-deterministic baked into the system prompt or an early message ($(date +%s), a random request ID, a UUID, an environment dump, a hostname, a PID) guarantees the prefix differs every run, which guarantees a miss every run.
// ❌ LOOKS INNOCENT — a request ID "for tracing," placed in the system prompt
const system = `${BASE_SYSTEM_PROMPT}\nrequest_id: ${crypto.randomUUID()}`;
// ❌ LOOKS INNOCENT — dumping the environment so the agent "has context"
const system = `${BASE_SYSTEM_PROMPT}\nenv: ${JSON.stringify(process.env)}`;
// ^^^ also a credential-leak waiting to happen// ✅ CORRECT — non-deterministic content belongs in the LAST (uncached) user turn
const userTurn = `request_id: ${crypto.randomUUID()}\n\n${actualMessage}`;
// the prefix stays byte-stable; the only thing that changes is the part that's supposed toThe principle generalizes: if it changes between runs, it goes in the last user turn, never in the prefix. The final turn is uncached anyway, so that’s the correct home for everything dynamic. As a bonus, keeping environment dumps out of the system prompt closes a credential-leak vector at the same time, which is why I treat this rule as security hygiene as much as cost hygiene.
Rules that only say “no” leave people guessing about the legitimate cases. There are exactly three things you are allowed to do, and between them they cover everything a real agent needs.
This is the whole point of the cache. You add a new user turn, the model responds, you append its assistant turn, and the prefix (everything before the new turn) is byte-identical to last time. So it all hits the cache. You pay full rate only on the new turn and ~10% on the warm history.
// The 95% case. The prefix is untouched; only the tail grows.
history.push({ role: 'user', content: nextMessage });
const reply = await call_llm({ system, tools, messages: history });
history.push({ role: 'assistant', content: reply });If your agent does nothing but this, you have already won the cache game. The forbidden mutations are all deviations from this happy path; the discipline is mostly about not deviating.
Sometimes you genuinely need a different system prompt, a different tool set, or a clean slate. That’s allowed, as long as you replace the whole conversation rather than mutating the existing one. A fresh session starts a fresh cache prefix. You expect a cache miss on the first turn, you keep nothing stale around, and you’re back on the append-only happy path from turn 2 onward.
The distinction that matters: replacing the whole conversation is fine; editing the existing one is the violation. New session, new prefix, clean. Same session, edited prefix, broken.
This is the sanctioned answer to “my context is getting too long.” You do not solve it by editing history (forbidden mutation #3). You solve it with a deliberate compress step that ends one conversation and seeds a new one with a summary.
Conversation A (long, getting expensive)
│
▼
[ compress ] ──► produce a summary of A
│
▼
Conversation B (fresh prefix; seed message = the summary)The compress step is the only sanctioned write path to the system prompt between conversations. It’s a clean boundary: conversation A is finished and discarded, conversation B starts fresh with a summarized seed, and B’s prefix is frozen from its own turn 1. Nothing in A’s prefix is ever mutated. You took the cache miss intentionally, at a moment you chose, instead of suffering it accidentally on every turn.
In my own daemon, this is what the handoff between sessions looks like. When a conversation has earned a compression, it ends, summarizes itself, and seeds the next one. It’s the difference between retiring a conversation and corrupting it.
Because this failure is silent, humans will eventually forget the rule. So I don’t rely on memory. I make the cheap-correct thing the default and make violations fail loud: before the code merges, and while it runs.
The first line of defense is a validator that runs in pre-commit and again in CI. It’s not clever. It greps the source for the forbidden patterns and fails the commit if it finds them:
system_prompt + …, template-literal interpolation into the base prompt, any reassignment of the captured prompt after first use.date, Date.now(), randomUUID, process.env dumps, PIDs, hostnames anywhere in the system-prompt construction path.tools.push(...), tools.filter(...), per-turn tool resolution inside the conversation loop instead of once at session start.history[i], .map() over the message array that returns mutated content, anything that writes back into an existing turn.A grep-based static check has obvious blind spots (it can’t catch a violation laundered through three layers of indirection) but it catches the overwhelming majority, which are exactly the “looks innocent” one-liners from the forbidden section. The goal isn’t a proof; it’s a tripwire on the common mistakes, sitting at the cheapest possible place to catch them: before they ever merge.
Static analysis can’t see what your code actually sends, so the second line of defense is a runtime audit that inspects the real request prefixes. It records the prefix each conversation sends and checks the invariant that matters most: within a single conversation, the system prompt and tool schema must be byte-identical across turns, and the message history must be a strict append of the previous turn’s history. If turn 5’s prefix isn’t a clean extension of turn 4’s, something mutated it, and the audit flags exactly where the divergence point landed.
This catches the cases grep can’t: the laundered mutation, the third-party library that reformats your prompt behind your back, the subtle reordering that only shows up at runtime. Static catches it cheap and early; runtime catches it true and complete. You want both.
Same honesty as the rest of this series: here are the places this discipline is genuinely hard, and the traps I’ve actually hit.
The single most common violation, and the most tempting, because it feels like good prompt engineering. You want the model to know the current task, the active file, today’s date, so you append it to the system prompt where the model “will definitely see it.” That append is a byte-zero cache miss every turn. The fix is muscle memory: dynamic context goes in the last user turn, which the model perceives with equal clarity and which doesn’t touch the cache.
Context gets long, tokens cost money, the obvious move is to trim old turns in place. It’s the wrong move: editing history is forbidden mutation #3, and it busts the cache for everything after the edit, so you spend more to “save” tokens. The right move is the compress step: end the conversation, summarize it, seed a fresh one. Editing corrupts; reseeding retires.
Narrowing the tool list to “what’s relevant this turn” feels like you’re helping the model focus. It isn’t: it busts the schema cache and degrades tool planning, a double cost. Decide the tool set once per session. If you truly need a different set, start a different conversation. (The full version of this trap is the entire MCP-in-production post, coming November 15; it’s the load-bearing constraint there.)
You can follow every rule and still get burned by a library that reformats your system prompt, injects a timestamp, or reorders tools “for you” between calls. This is exactly why the runtime audit exists: static checks only see your code. If your cache-hit rate is mysteriously bad and your source looks clean, audit the actual bytes on the wire. The mutation is usually something you didn’t write.
Cache discipline is invisible when it works and brutally expensive when it doesn’t. The model stays fine while your bill silently climbs. Treat it as an invariant, not a tip.
The prefix is append-only and frozen. Everything dynamic goes in the last user turn. The four forbidden mutations are all the same violation, reaching back into the prefix, wearing four masks: system-prompt edits, tool-schema changes, history rewriting, and non-deterministic content.
At a $20/month budget, the cost ceiling is what lets the agent be autonomous. Cache discipline is what holds that ceiling. Break it and you don’t get “a pricier agent.” You get an agent that trips its own budget kill switch and stops being autonomous at all. Cheap is the feature.
In this series:
For developers
Pick one agent loop you run. Check three things: is the system prompt byte-identical every turn, is the tool set decided once per session, and is the history strictly append-only? If any answer is 'no,' you have a silent cost bug. Add the grep tripwire, then read Part 2 for the full economics.
Running an agent on a tight budget, or fighting a mystery bill? Tell me where your cache-hit rate went and what you found when you audited the bytes on the wire. The silent failures are the ones worth comparing notes on.