The optimization that’s bigger than people think
Prompt caching can cut inference cost by 50–90% on the right workloads. It is one of the largest single cost optimizations available on frontier providers, and most production systems either don’t use it or use it badly.
The reason it’s underused: the documentation makes it sound complicated, the savings depend on workload shape in ways that aren’t obvious, and the first time a team tries it the savings underwhelm because the cache structure was wrong. So they decide it doesn’t work and move on.
It works. The workloads where it works just have to match a specific shape, and the implementation has to respect a few rules. Below is what we’ve learned shipping it across a dozen production systems.
The mechanics, briefly
The frontier providers — most prominently Anthropic, with GPT and Gemini catching up — let you mark portions of your prompt as cacheable. The first time a marked prefix is sent, the provider stores the processed representation. Subsequent calls that share the same prefix get a discount: typically 10% of normal input cost for cached tokens vs the full rate.
The cache is keyed by the exact prefix. Any change in the first character invalidates the cache. The cache has a TTL (typically 5 minutes to an hour depending on provider tier) and is per-organization.
That’s the entire mechanism. The interesting question is when it pays off.
The workload shapes where caching wins big
Caching wins when:
- The same long prefix gets reused across many calls.
- The variable suffix is short relative to the cached prefix.
- The calls happen often enough to keep the cache warm.
Three specific shapes match this profile, and they happen to be common in agent systems.
1. Workflow + toolset descriptions
Every call to an orchestrator agent sends the same workflow document and toolset description in the system prompt. Often 3,000–8,000 tokens of stable content. The variable part — the current state and the model’s response — is a few hundred tokens.
This is the canonical caching win. Cache the workflow + toolset as the prefix. For an active agent handling 100 calls/hour, cache hits drop the per-call cost by 60–80% on input tokens.
2. RAG context with stable document sets
When you retrieve the same documents repeatedly (because they’re the canonical reference for a category of question), the retrieved context is cacheable.
Less obvious win, but real. For a customer support agent that draws on the same 50 product-doc chunks for most queries, caching the retrieved context layer saves substantially.
3. Long static reference documents
Tasks that operate on a single long document — codebase analysis, legal contract review, transcript summarization — cache the document and vary only the query.
For a long codebase analysis agent that processes 100 different queries against the same 200k-token codebase, caching cuts cost by 70%+.
The workload shapes where caching doesn’t help
Caching doesn’t help when:
- The prefix changes between calls (per-user customization, per-tenant variation).
- The cacheable prefix is short relative to the total prompt.
- Calls are too sparse to keep the cache warm.
Common shapes that don’t benefit:
- Per-user fine-tuned prompts. Different prefix per user → no cache reuse.
- Tightly customized per-tenant agents. Tenant config in the prefix invalidates cross-tenant caching.
- Low-volume infrequent calls. The cache TTL expires between calls.
- Workloads dominated by output tokens. Caching only affects input tokens; output cost is unchanged.
If your workload looks like these, caching is decoration, not optimization. Don’t waste the engineering time.
The architectural rules
When caching does apply, get the structure right:
1. Stable content goes first
Cache markers attach to a prefix. Everything from token 0 to the cache marker is what gets cached. So put your most stable content first, your most variable content last.
Order: system instructions (stable) → workflow doc (stable) → toolset description (stable) → cached marker → tenant config (semi-stable) → conversation history (variable) → current message (most variable).
2. Cache markers at multiple levels
Most providers allow multiple cache breakpoints. Use them. A typical setup:
- Cache marker after system instructions + workflow.
- Cache marker after toolset description.
- Cache marker after tenant config.
- No cache marker for the conversation tail.
This way, the most stable content is cached across all calls, the moderately stable content is cached across many calls, and you avoid invalidating the entire cache when something small changes.
3. Stable means stable
If you suspect a portion of your prompt might change, don’t put it in the cached prefix. The cost of cache misses (full input cost, plus the storage write) is higher than the cost of just not caching that portion.
Concretely: timestamps, user names, dates that change per call — these are not cacheable.
4. The cache write costs slightly more
The first call that populates the cache costs ~125% of a normal call (the cache write surcharge). Subsequent calls cost ~10% of input cost on cached tokens. The break-even is around 2 calls. Below that, caching costs you money.
For production workloads, this is irrelevant — you’re well past 2 calls. For low-volume agents, it’s a real consideration.
What this looks like in cost math
A concrete example. An orchestrator agent with:
- System + workflow + toolset = 6,000 tokens (cacheable).
- Current state + conversation = 1,500 tokens (not cacheable).
- Total input: 7,500 tokens per call.
- Volume: 50 calls/hour, 1,200 calls/day.
Without caching:
- 1,200 calls × 7,500 input tokens × $3/MTok = $27/day.
With caching (95% cache hit rate after warm-up):
- 60 cache writes × 6,000 tokens × $3.75/MTok = $1.35/day.
- 1,140 cache hits × 6,000 tokens × $0.30/MTok = $2.05/day.
- 1,200 calls × 1,500 variable tokens × $3/MTok = $5.40/day.
- Total: $8.80/day.
The cached version is 67% cheaper on input cost. Across an active agent system, this compounds quickly.
The mistakes we see most often
Three patterns that destroy caching benefit:
1. Putting a timestamp in the system prompt. “It is currently 2026-01-09 14:32 UTC.” This invalidates the cache on every call. Wide error mode; small to fix; do it.
2. Including the user’s name in the system prompt. Same pattern. Move user identity past the cache marker.
3. Embedding small variable values throughout a long prefix. A common pattern is interpolating user-specific things into the long context. Each interpolation invalidates the cache. The fix: move variables to the end, parameterize the prompt structure to keep stable content stable.
The pattern: anything that varies has to be downstream of the last cache marker. Anything that’s stable has to be upstream.
When to revisit caching
Cache strategy isn’t set-and-forget. Re-evaluate when:
- Your workflow doc changes. The cache is invalidated; you may want to redesign which parts are stable vs variable.
- Your toolset changes. New tools added to the description invalidate caching.
- Your model provider updates pricing. Caching economics depend on the discount ratio; check quarterly.
- You add per-tenant customization. Most tenant customization breaks shared caching; consider whether the per-tenant content is worth its cost.
In practice this is a quarterly review for active systems.
What caching won’t do for you
Two limits to manage expectations:
Caching doesn’t reduce output token cost. Output is unchanged. If your workload is output-dominated, caching is a small portion of the optimization story.
Caching doesn’t substitute for prompt engineering. A bloated 12,000-token prompt cached cheaply is still worse than a 4,000-token prompt that does the same job. Get the prompt right first, then optimize with caching.
The take
Prompt caching is one of the largest cost optimizations available on frontier providers, and one of the most underused. The workloads where it wins are common in agent systems — long stable prefixes with short variable tails. The architecture rules are simple — stable content first, multiple cache markers, no variables in the prefix. Get this right and you cut your inference bill by half or more on the right workloads. Get it wrong and the savings underwhelm and the team gives up. The math is real. The implementation matters.
Prompt caching audits are part of the cost-optimization review in our Custom AI Builds. Schedule a call if you suspect your current caching strategy is leaving money on the table.
