Prompt caching stores the key-value tensors computed during prefill so that subsequent requests beginning with the same prefix skip reprocessing those tokens. The savings are substantial: Anthropic charges cache reads at one-tenth of its standard input price — a 90% discount — and OpenAI bills cached tokens at a reduced input rate across eligible models. For workloads that repeat system prompts, tool schemas, or retrieved context across calls, prompt caching is the fastest cost reduction available without changing models.
How Prompt Caching Works
During the prefill phase of transformer inference, the model computes attention key-value tensors for every input token. Prompt caching persists those tensors so that a later request with an identical prefix can reuse them, skipping the compute-bound portion of the forward pass. The mechanism is strictly prefix-based: only content from the start of the prompt that matches character-for-character between requests qualifies for a cache hit. A single changed character in the system prompt, a reordered tool definition, or a timestamp injected at the wrong position invalidates the cache from the divergence point forward.
Both major API providers gate caching behind a minimum prefix length. OpenAI requires at least 1,024 tokens in the cached prefix before any caching occurs; shorter prefixes process normally with no discount. Anthropic applies a minimum that varies by model tier. The prefix boundary is the single most important constraint: everything above it must be identical across requests, and everything below it can vary freely.
OpenAI vs Anthropic Caching
The two providers differ in how they expose caching controls and how they price cache writes and reads. The table below summarizes the key engineering differences.
| Dimension | OpenAI | Anthropic |
|---|---|---|
| Enablement | Automatic for 1024+ token prefixes; no code change required | Automatic top-level cache_control or explicit breakpoints on blocks |
| Cache read pricing | Discounted input rate on cached tokens | 10% of base input price (90% discount) |
| Cache write fee | Free (pre-GPT-5.6); 1.25x input rate (GPT-5.6+) | 1.25x base (5-min TTL); 2x base (1-hour TTL) |
| Default TTL | 30 minutes (GPT-5.6+); 5-10 min in-memory (older) | 5 minutes, refreshed on each cache hit; 1-hour available |
| Routing control | prompt_cache_key parameter for affinity | No explicit routing key; prefix hash determines placement |
Anthropic publishes an explicit per-model pricing table that makes the cost math transparent. On GPT-5.6 models and later, OpenAI charges cache writes at 1.25x the uncached input token rate, while earlier model families impose no additional write fee. This means the break-even analysis differs by model generation: on pre-GPT-5.6 OpenAI models, caching is profitable if the prefix is reused even once within the window; on GPT-5.6+, you need enough cache reads to offset the write premium.
Structuring Prompts for Cache Hits
The single most important engineering decision is prompt ordering. Because caching is prefix-based, static content must come first and dynamic content last. This is not a recommendation — it is the mechanism itself. The ordered procedure below maximizes cache reuse:
- Place system instructions, role definitions, and output-format schemas at the very top of the prompt.
- Add tool definitions and function-calling schemas next, as these rarely change between calls.
- Append any retrieved context documents or few-shot examples after the tools block.
- Put user-specific data — the actual query, timestamps, session identifiers — at the very end.
Reordering these blocks breaks caching. A system prompt that includes a timestamp or a per-user greeting at its start will never match a cached prefix, because that content changes with every request. The fix is mechanical: extract variable content into the final message and keep everything above it byte-identical across calls in the same session. On OpenAI GPT-5.6 models and later, you can additionally set a prompt_cache_key parameter that routes requests sharing a long common prefix to the same serving machine, improving hit rates. The provider recommends keeping total traffic per key at approximately 15 requests per minute and partitioning across additional keys for higher-volume workloads.
Monitoring and Tuning Hit Rates
Both APIs report cache usage in their response objects. OpenAI returns cached_tokens inside usage.prompt_tokens_details, showing how many input tokens were served from cache. Anthropic returns cache_read_input_tokens and cache_creation_input_tokens in the usage block. Logging these fields on every request is the only reliable way to verify that caching is actually working in production.
A practical monitoring checklist for inference teams:
- Log cached token counts per request and compute the daily cache hit ratio (cached tokens divided by total input tokens).
- Set alerts for hit-ratio drops below your workload baseline — these usually indicate prompt drift after a deployment.
- Compare cache-write spend against cache-read savings weekly to confirm net positive ROI.
- Verify prompt structure changes do not regress hit rates before promoting to production.
- For low-frequency workloads, evaluate whether a 1-hour TTL (Anthropic) or extended retention (OpenAI) justifies the higher write cost.
For teams running self-hosted models, the same prefix-reuse principle applies through vLLM automatic prefix caching combined with continuous batching, which together reuse KV cache tensors across requests with shared prefixes. For a broader view of where caching fits in the LLM cost stack, see our analysis of FinOps for AI inference.