Provider prompt caching is a discount the model providers themselves offer on the repeated, static portion of your prompts – roughly 50% off cached input tokens on OpenAI, 90% on Anthropic, and 75% on Gemini.
Unlike a gateway-side semantic cache (which skips the call entirely), provider prompt caching makes the calls you do send cheaper – it lets the provider reuse the computation for prompt prefixes it has seen before. The two stack.
How provider prompt caching differs from semantic caching
These are two different mechanisms people often mix up. A semantic cache sits in front of the provider and skips the call when it can serve a stored answer. Provider prompt caching sits at the provider and discounts the call you still make – when your prompt shares a prefix with a recent request, the provider reuses the key-value computation for that prefix instead of recomputing it, and charges you a fraction of the normal input price for those cached tokens. You want both: semantic caching to skip calls, provider prompt caching to cheapen the ones you make.
 | Semantic cache (gateway) | Provider prompt cache |
Where it runs | In front of the provider | At the provider |
What it saves | The entire call | Input cost on the repeated prefix |
Trigger | Similar request seen before | Same prompt prefix seen recently |
Typical benefit | 100% on a hit | ~50/90/75% off cached input |
Who controls it | You (the gateway) | The provider, via hints you send |
Each provider does it differently
The mechanics vary by provider, which is exactly why automatic hint injection earns its place – application teams shouldn’t have to learn four different caching APIs.
- Anthropic – explicit cache_control breakpoints you place in the request. You get ephemeral (5-minute, no write premium) and extended (1-hour, with a write premium) options. The deepest discount of the bunch: ~90% off cached input.
- OpenAI – a prompt_cache_key plus automatic prefix caching. ~50% off cached input tokens.
- Gemini – cachedContents with configurable TTLs (5 minutes, 1 hour, 6 hours, 24 hours). ~75% off cached input. It has a higher minimum-token threshold before caching kicks in.
Bedrock – cachePoint markers for the models that support them.
Automatic prefix marking
The savings come from marking the right part of the prompt as cacheable – the static prefix that repeats across requests. That’s usually the system prompt, the tool definitions, and any large static content blocks. A gateway can inject these markers for you: detect the provider, find the stable prefix, and apply the provider-correct cache hint. A minimum static-token threshold (often around 1,024 tokens, higher for Gemini) keeps it from marking prefixes too small to be worth caching.
Why TTFT reordering compounds with this
Provider prompt caching only helps if the prefix is genuinely stable across requests. If your message ordering wanders – sometimes system-first, sometimes context-first – the prefix changes and the cache misses. That’s why prefix reordering (keeping system messages and older turns in a stable, canonical order at the head) compounds with provider caching: reordering maximizes the stable prefix, which maximizes the cache-hit rate, which maximizes the discount. The two layers reinforce each other.
🔗 Internal link: Link ‘prefix reordering’ to the TTFT discussion – note for Batch 2 this is covered inline in Post 9’s stack; cross-link to Post 9. Link ‘semantic cache’ to Post 10.
A rollback safety property
One practical concern: if the gateway injects cache markers and you later want to turn provider caching off, you don’t want to go re-edit application code. A well-designed gateway strips its own cache markers when the operator disables the feature – so caching rolls back with a config switch, not a code change. The application keeps sending requests unchanged; the gateway decides whether the markers go through.
🔗 Internal link: Primary CTA: /platform/cost-optimization/. Link up to Post 9 (pillar).
How DeepintShield approaches this
DeepintShield injects provider-cache hints for you: it detects the target provider and marks the stable prefix with the provider-correct mechanism – Anthropic cache_control, OpenAI prompt_cache_key, Gemini cachedContents, Bedrock cachePoint – above a minimum token threshold, and strips its own markers cleanly if you turn the feature off. Because it also reorders messages to maximize the stable prefix, the cache-hit rate (and the discount) goes up. For teams that don’t want application developers learning four different caching APIs, DeepintShield is one way to capture provider prompt-cache discounts through a single gateway.
