Fail over between LLM providers
Try backup providers in your order, balance traffic across keys and remove failing keys with a circuit breaker, at the gateway.

Contents
Learning how to fail over between LLM providers usually means writing retry logic into every application. DeepintShield does it once, at the gateway: it tries backup providers and models in the order you set, spreads traffic across keys, and removes failing keys with a circuit breaker, all behind one OpenAI-compatible API.
The challenge
Every model provider has outages, rate limits and slow periods. Handling them in application code causes its own problems:
- Different SDKs, different errors. Each provider signals failure in its own way, so retry code multiplies.
- Rate limits hit at peak. HTTP 429 responses arrive exactly when traffic matters most.
- Silent quality changes. A fallback model may answer differently, and nobody notices which model served a request.
- Controls get skipped. Hand-written retries can bypass logging, budgets and guardrails.
The approach
Define fallback chains centrally and test them. Treat every attempt as a full request, so the same governance applies. Distinguish errors worth retrying from requests that were refused on purpose. Spread load across several keys and stop sending traffic to keys that are failing. Record which provider and key served each request so quality and cost can be checked afterwards. Keep chains short: every extra attempt adds latency and cost when a provider is struggling. Decide in advance which models are acceptable substitutes for each task, and which tasks should fail rather than fall back.
How to fail over between LLM providers with DeepintShield
- Define fallbacks. Send a
fallbackslist in the request (in Python, throughextra_body), or give a Virtual Key several providers so it builds a chain in weight order when the request has none. Fallbacks trigger on network errors, HTTP 500, 502, 503 and 504, 429, model unavailable, timeouts and authentication failures. They do not trigger on validation errors or guardrail and governance blocks. Each attempt is a fresh request, so caching, governance and logging apply again. See the fallbacks docs. - Spread load across keys. Give each provider key a weight; the gateway uses weighted random selection and moves to the next eligible key if one fails. Enable the circuit breaker with
load_balancer_enabledto remove failing keys and retry them periodically. On Team and above, adaptive load balancing shifts traffic by error rate, latency and recovery, with key states Healthy, Recovering and Circuit Open. See adaptive load balancing. - Watch which provider served. The serving provider is returned in
extra_fields.provider. Prometheus labels include the selected key and the fallback index. If every attempt fails, the original primary error is returned. Alert on the fallback rate as well as the error rate.
The gateway does not route automatically across providers for a model you did not configure, so list fallback models explicitly and check their answer quality for your task.
What you can show afterwards
- Which provider and key served each request, and how often fallbacks ran.
- Key health over time, including when keys entered Circuit Open and recovered.
- Error rates by provider, ready for a capacity or vendor review.
- Governance and logging records for fallback attempts, the same as for first attempts.
Run a failover drill before you need it. Force a timeout or errors on the primary key, confirm the fallback serves the request, check that the response names the serving provider, and find the fallback index in your metrics. Then compare a sample of fallback answers with primary answers for quality.
Planning a move to a gateway? Read why migration is more than a base URL change.
Related
- AI Gateway: routing, fallbacks and load balancing
- AI security for banks: resilience for customer-facing AI
- Give every team safe LLM access: one route to every provider
- Set LLM budgets for each team: over-budget providers drop out of routing