Skip to content

Meet DeepintShield at Bengaluru Tech Summit 2026, 17 to 19 November, BIEC Hall 1, booth H1A106. Book 20 minutes

Start For Free

Platform · Cost Optimization

LLM cost optimization: 12 techniques in one gateway

LLM cost optimization lowers what you pay for model usage while keeping answers fit for purpose. DeepintShield applies 12 techniques on the request path, from provider prompt caching to semantic caching and cascade routing. Semantic caching can cut costs by up to 60% on repetitive workloads; your savings depend on your workload.

  • 5 techniques on every plan
  • 7 more on Team and above
  • No token markup
LLM cost optimization illustration: a stack of layers with a downward arrow and a coin

Why one cost tactic is not enough

Model spend grows from several kinds of waste at once. The AI gateway sees all of it, so that is where the fixes belong.

Different waste, different fixes

Repeated questions, long prompts, growing histories and oversized models each need a different control.

Savings can hide quality loss

A cheaper model or a shorter prompt can lower answer quality if nobody measures the result.

Spend is hard to attribute

Without figures per key, team and customer, finance cannot see where model spend goes.

How LLM cost optimization works

Each technique targets one kind of waste. See how a request moves through DeepintShield.

Reuse paid-for work

Exact and semantic caches, provider prompt caching and request coalescing avoid repeated work.

Right-size the model

Cascade routing tries a cheaper model first; reasoning-effort controls stop unneeded reasoning.

Trim the input

Prompt compression, RAG optimization and conversation summarization cut tokens before the call.

Measure the result

Prometheus metrics and the console show cost, cache hits and savings by key and team.

The 12 cost techniques, by plan

Five techniques run on every plan. Seven more, including semantic caching, need the Team plan or above. A request served from cache skips the later steps, so measure the combined bill rather than adding up percentages.

The 12 cost techniques, by plan
TechniqueWhat it reduces
Every plan, including Developer
Provider prompt-cache hintsRepeated prompt prefixes, billed at the provider's cached-input rate
Request coalescingIdentical concurrent requests, which share one upstream call
TTFT prefix orderingPrompt changes that stop a provider cache from matching
Guardrail evaluation cacheRepeated deterministic guard checks, reused in about 1.42 µs (local microbenchmark, no I/O)
Parallel-tool safetySerial waits for independent tool calls
Team plan and above
Semantic cachingRepeated or similar requests, answered from an exact or vector cache
Cascade routingUse of a stronger model when a cheaper one is good enough
Reasoning-effort throttlingReasoning beyond what the task needs
Prompt compressionRedundant input text
RAG optimizationIrrelevant retrieved passages in the context
Conversation summarizationLong histories sent again on every turn
Batch eligibility analysisNon-urgent work that could run as a batch; it reports eligibility and does not submit batches

Team and above can also opt in to MCP tool-result caching. Budgets and rate limits per key and team are part of Governance.

Inside the main savings techniques

Exact match first, then similarity

An exact hash lookup runs first, then vector similarity with a default threshold of 0.88. Entries carry the guardrail fingerprint and full request scope, so a policy change misses the cache. Works with Weaviate, Redis or Valkey, Qdrant and Pinecone.

Semantic cache for LLM cost optimization: an exact hash lookup first, then vector similarity at 0.88 by default; on a miss the model answers and the result is stored with its guardrail fingerprint(opens the full-size image in a new tab)

Cache hints added for you

On by default for Anthropic, OpenAI and Bedrock, and you can add Gemini. The gateway marks stable prompt prefixes and forwards cache hints, with no application change. Provider pricing gives roughly 90% off cached input on Anthropic, 50% on OpenAI and 75% on Gemini.

Automatic provider prompt cache hints added to stable prompt prefixes(opens the full-size image in a new tab)

Cheaper model first

A cheaper model answers first, and the request escalates to a stronger model when confidence is low. The cascade checks the key's model policy before it substitutes a tier.

Cascade routing with a confidence gate: cheaper model first, escalation when confidence is low(opens the full-size image in a new tab)

One request path

Each technique sits at a fixed point on the request path, so a request served from cache never reaches compression or routing.

Where each cost optimization layer sits on the LLM request path(opens the full-size image in a new tab)

Cost controls in one settings page

Switch provider prompt caching and semantic caching on and tune them in one place, then follow cost, cache hits and savings on the analytics overview. Prometheus exposes deepintshield_cost_total and deepintshield_cache_hits_total for your own dashboards.

  • Per-request cache headers for TTL, threshold and opt-out
  • Agentic Cache reporting on Business and above
  • No token markup and no logging surcharge
DeepintShield Cost Optimization settings: provider prompt caching for four providers, and semantic caching with its embedding model(opens the full-size image in a new tab)
Cost Optimization settings in the DeepintShield console.

Turn on savings without code changes

Point your existing client at the gateway and the caching techniques apply to its traffic. When one request needs different cache behaviour, set it with a header.

import openai

client = openai.OpenAI(
    base_url="https://app.deepintshield.com/openai",  # Point to DeepintShield
    api_key="sk-ds-your-virtual-key"
)

Works with your providers and stores

Works with

  • 29 built-in model providers
  • Anthropic, OpenAI, Gemini and Bedrock prompt caching
  • Weaviate
  • Redis and Valkey
  • Qdrant
  • Pinecone
  • Prometheus

Semantic caching needs the Team plan or above, a vector store and embeddings configured.

Further reading on reducing LLM spend

LLM cost questions

How much can I save on LLM costs?

It depends on your workload. Semantic caching can cut costs by up to 60% on repetitive workloads such as FAQ assistants, while long-form RAG or streaming code generation sees much less. The semantic caching docs explain what drives the hit rate.

Does caching affect answer quality?

It can, which is why cache scope is strict. Entries carry the guardrail fingerprint and the full request scope, structured, multimodal and tool requests are exact-match only, and semantic hits still pass output checks.

What is semantic caching?

Semantic caching returns a stored answer when a new request means the same as an earlier one. DeepintShield checks an exact hash first, then vector similarity at a default threshold of 0.88, and keeps entries separate for each virtual key.

What is cascade routing?

Cascade routing sends a request to a cheaper model first and escalates to a stronger one only when needed. It is available on Team and above, and it never substitutes a model that the key's model policy forbids.

Is there a markup on model usage?

No. You bring your own provider keys, model usage stays on your own accounts, and there is no token markup or logging surcharge. See pricing.

Lower your LLM bill without losing control

Start with five techniques on the free Developer plan, then add semantic caching and cascade routing on Cloud Team.

Start For Free