Back to blog

Prompt Caching Is Everything in Claude Code

12 min read

I've been using Claude Code heavily for the past few months - building projects, debugging code, writing features. Some sessions felt snappy and cheap. Others felt sluggish and expensive, even when I was doing similar work. I couldn't figure out why until I dug into how prompt caching actually works.

Turns out, caching isn't just a nice optimization. It's the thing that makes long-running AI coding agents economically viable. Anthropic's own team has said they build their entire harness around prompt caching, run alerts on cache hit rates, and declare incidents when the rate drops too low.

Once I understood how it works, I changed how I use Claude Code. Here's what I learned.

What's Actually Happening Under the Hood

Every time you send a message in Claude Code, it makes an API call. That call includes everything: the system prompt, all tool definitions (bash, file edit, search, etc.), your CLAUDE.md files, and your entire conversation history. On a long session with a big codebase loaded, this can easily be 50,000 to 200,000+ tokens of input.

Without caching, Anthropic's servers process all of those tokens from scratch on every single message. Even though 99% of them are identical to what was sent 30 seconds ago. That's slow and expensive.

With caching, the server saves its work on the unchanged portion. The next message only needs to process the new content - your latest question and Claude's last response. Everything before that is served from cache at 1/10th the cost.

The math is dramatic. On a 20-turn session with 100K tokens of stable context:

  • Without caching: 20 x $0.30 = $6.00 in input costs
  • With caching: $0.375 (first write) + 19 x $0.03 = $0.945
  • Savings: 84%

And it's not just cost. Cached tokens are served significantly faster - time-to-first-token drops substantially.

How Prefix Matching Works

This is the key concept. Caching works by prefix matching - it caches everything from the start of the request forward, and stops at the first byte that's different from what it cached last time.

Think of the API call as a stack:

[Tool definitions: ~5K tokens]     ← static, always the same
[System prompt + CLAUDE.md: ~3K]   ← static within a session
[Turn 1 Q&A: ~550 tokens]         ← grows each turn
[Turn 2 Q&A: ~600 tokens]         ← grows each turn
[Your new question: ~50 tokens]    ← new content

The cache matches from the top down. On turn 3, it finds that tools + system prompt + turn 1 + turn 2 question are all identical to what it cached. So it serves all of that from cache. Only turn 2's answer and your new question are processed fresh.

By turn 5, you might have 95%+ cache hits. By turn 20, it's even higher.

But here's what makes it fragile: the cache breaks at the first point of difference and everything after that point is processed fresh, even if it's identical to what was cached. It's not a key-value cache. It's a prefix match. Any change anywhere in the prefix invalidates everything after it.

[cached] [cached] [cached] [CHANGED] [not cached] [not cached]
                               ↑
                        Cache stops here.
                  Everything after is processed fresh,
                  even if identical to last time.

The Things That Break Your Cache

Once I understood prefix matching, a lot of mysterious behavior started making sense.

Starting a new session. Every new session starts with a cold cache. No cache carries over between sessions. This is why one long session is cheaper than three short ones for the same work.

Switching models. Caches are isolated per model. If you switch from Sonnet to Opus mid-session with /model, your entire Sonnet cache is useless. Opus starts from scratch. I was doing this constantly - using Sonnet for simple tasks and switching to Opus for complex ones - and it was killing my cache hit rate every time.

The right approach if you need a different model for a specific task: let the agent handle it with subagents. Claude Code already does this with Explore agents that use Haiku. The main conversation's cache stays intact because the subagent runs in a separate context.

Toggling features. Turning web search or extended thinking on/off changes the system prompt or tool definitions, which sit at the top of the cache hierarchy. Change those and everything below loses its cache.

Using /rewind. This one surprised me. Rewinding even a single turn changes the prefix - the cached version includes the message you removed, but your new request doesn't. If you've built up 30 turns of context and rewind, you lose the cache on everything after the rewind point.

Letting the timer expire. Cached content expires after a period of inactivity. Pro plan users get 5 minutes. Max plan users get 1 hour. Every time you hit the cache, the timer resets. So as long as you keep working, it stays warm. But step away for a coffee break on a Pro plan and your cache is gone.

What Doesn't Break the Cache

This is equally important:

  • Adding a new message at the end - the prefix is preserved
  • Claude giving different-length responses - output is never cached, only input
  • Reading files into context - they're appended as messages, not injected into the system prompt

The Design Decisions That Blew My Mind

The Anthropic team made some counterintuitive architectural choices in Claude Code that only make sense once you understand caching. These are genuinely clever.

Plan Mode Doesn't Change Tools

The obvious way to implement plan mode: when the user enters plan mode, swap out the tool set to only include read-only tools. Makes total sense from a safety perspective.

But that would break the cache. Changing tools changes the prefix, and everything after is invalidated.

Instead, Claude Code keeps all tools in the request at all times. Plan mode is implemented as a tool itself - EnterPlanMode and ExitPlanMode. When you toggle plan mode, the agent gets a system message explaining the constraints. The tool definitions never change.

Bonus: because EnterPlanMode is a tool, the model can autonomously enter plan mode when it detects a hard problem. No cache break, no user intervention.

Tool Search Uses Stubs Instead of Removal

Claude Code can have dozens of MCP tools loaded. Including all of them in every request would be expensive. But removing them mid-conversation would break the cache.

The solution: lightweight stubs. Instead of removing tools, Claude Code sends just the tool name with a defer_loading: true flag. The model can "discover" full tool schemas via a ToolSearch tool when needed. The stubs are always present in the same order, so the cache prefix stays stable.

Compaction Shares the Parent's Prefix

When you hit the context window limit, Claude Code summarizes the conversation and starts fresh. The naive approach would be: make a separate API call with a different system prompt and no tools to generate the summary.

That would be a complete cache miss. You'd pay full price for all those tokens.

Instead, Claude Code uses the exact same system prompt, tools, and conversation history as the parent conversation. It appends the compaction prompt as a new message at the end. From the API's perspective, this looks nearly identical to the parent's last request - same prefix. The cached content is reused. The only new tokens are the compaction prompt itself.

System Reminders Instead of Prompt Updates

Sometimes information in the system prompt becomes outdated - the time changes, a file gets modified. The tempting approach: update the system prompt. But that breaks the cache from the system prompt onward, which is near the top of the stack.

Instead, Claude Code inserts <system-reminder> tags in the next user message or tool result with the updated information. "It is now Wednesday." The system prompt stays unchanged, the cache stays intact, and the model still gets the updated information.

How I Changed My Workflow

After understanding all of this, I made some concrete changes:

I stopped switching models mid-session. If I start with Sonnet, I stay with Sonnet. If I need Opus for something complex, I'll start a new session or let the agent use subagents.

I batch related work into one session. Loading a large codebase into context and asking 5 questions in one session is dramatically cheaper than 5 separate sessions. One cache write versus five.

I stopped using /rewind. If I want to try a different approach, I fork the session instead of rewinding. Forking preserves the cache on the shared prefix. Rewinding destroys it.

I use session forking for parallel investigations. Build a base session with shared context (load files, establish the problem), then fork into parallel investigations:

bash
# Build shared context
claude
> Read src/db/queries.ts, src/api/routes.ts, src/middleware/auth.ts

# Fork into parallel investigations
claude --resume base-session --fork-session
> "Investigate the database timeout errors"

claude --resume base-session --fork-session
> "Check if token expiry handling could cause the 401s"

The base session pays the cache write once. Each fork gets cache hits on the shared context.

I keep sessions alive during active work. On the Pro plan with a 5-minute TTL, I make sure I'm sending messages regularly during focused work. If I need time to think, I accept that the next message will rebuild the cache - it's a one-time cost.

The Broader Lesson for Agent Builders

If you're building any kind of long-running agentic product, prompt caching should be a first-class architectural concern, not an afterthought. The Claude Code team puts it this way: "Cache rules everything around me."

The principles:

  1. Static content first, dynamic content last. Maximize prefix sharing across requests. Tool definitions and system prompts at the top, conversation messages at the bottom.

  2. Never change the prefix mid-session. Don't add or remove tools. Don't change the system prompt. Use messages to communicate updates instead.

  3. Design features around the cache, not the other way around. Plan mode, tool search, compaction - all designed to avoid cache breaks. If a feature requires changing the prefix, find another way to implement it.

  4. Monitor cache hit rate like you monitor uptime. The Claude Code team alerts on cache breaks and treats them as incidents. A few percentage points of cache miss rate can dramatically affect cost and latency.

  5. Prefer forking over restarting. When you need parallel workstreams or want to try a different approach, fork the session. Restarting or rewinding destroys the cache.

The thing that struck me most is how many of Claude Code's design decisions are driven by caching constraints rather than by what seems "obvious" from a product perspective. The obvious implementation often breaks the cache. The cache-aware implementation often seems weird until you understand why.

This is a pattern I've seen before in other systems. At scale, the performance constraint becomes the design constraint. You don't build the ideal architecture and then optimize it for caching. You build around the cache from day one.


I'm still learning new things about how Claude Code works under the hood. If you've found other patterns that improve cache hit rates, I'd love to hear about them.

aiclaude-codeperformancecachingengineering