Cascade routing sends each request to a cheap model first and escalates to a more expensive one only when the cheap model’s answer doesn’t clear a confidence threshold – routing the majority of low-stakes traffic to cheaper models without sacrificing quality on the requests that need a stronger one.
It’s built on a simple observation: not every request needs your best model, but you can’t tell in advance which ones do.
The waste cascade routing eliminates
Most teams pick one model for a workload and send everything to it. Go premium to be safe, and you overpay on the majority of requests a cheaper model would have handled perfectly. Go cheap to save money, and you get poor answers on the minority of hard ones. Neither is right, because the workload is mixed: most requests are easy, a few are hard, and they’re interleaved unpredictably. Cascade routing breaks the deadlock – try cheap, escalate only when needed.
Picking one model for a mixed workload means either overpaying on the easy majority or underperforming on the hard minority. Cascade routing refuses the tradeoff: cheap by default, strong on demand.
How confidence gating works
The mechanism hinges on judging whether the cheap model’s answer is good enough. Three confidence sources are common:
- Log-probability – the model’s own token probabilities. Low confidence (high uncertainty) on the generated answer means the cheap model is struggling – so escalate.
- Self-evaluation – ask the cheap model (or a cheap judge) to rate its own answer. Below a set bar, escalate.
Schema validation – for structured outputs, check whether the cheap model’s answer parses and conforms. If it doesn’t, escalate to a model more likely to nail the structure.
The parallel-mode variant
A sequential cascade (cheap, then strong if needed) adds latency on the requests that escalate – you spend the cheap call’s time before starting the strong one. For latency-sensitive workloads, parallel mode fires both models at once and keeps the cheap answer if it clears the threshold, throwing away the strong one. You pay for both calls on escalated requests (so it’s less cost-efficient), but you never add latency. It’s a cost-versus-latency call you make per workload.
Picking the model pair
Cascade routing works best when there’s a wide capability-and-price gap between the cheap and strong models – a small fast model (GPT-4o-mini, Claude Haiku, a Llama on Bedrock) as the first tier, a premium model as the escalation target. The bigger the price gap, the more each avoided escalation saves. On typical mixed workloads, 60–80% of requests get handled by the cheap tier, so most of your traffic runs at a fraction of premium pricing while the hard minority still gets the strong model.
Cascade routing vs reasoning throttling
These two are cousins that work on different axes. Reasoning throttling tunes how much a single reasoning model deliberates. Cascade routing picks between different models of different strength. They compose: cascade to choose the right model, then throttle that model’s reasoning effort to the task. Together they make sure you’re using neither too strong a model nor too much reasoning for the work in front of you.
🔗 Internal link: Link ‘reasoning throttling’ to Post 13. Link up to Post 9 (pillar).
🔗 Internal link: Primary CTA: /platform/cost-optimization/.
How DeepintShield approaches this
DeepintShield does cascade routing with the confidence gating described here: it tries a cheap model first and escalates to a stronger one only when confidence (logprob, self-evaluation, or schema validation) falls below threshold, with an optional parallel mode for latency-sensitive workloads. Paired with reasoning-effort throttling, it makes sure you’re using neither too strong a model nor too much reasoning for the task. For teams running mixed workloads where most requests are easy but a few are hard, DeepintShield is one way to route the majority to cheaper models without sacrificing quality on the rest.
