Skip to content

Meet DeepintShield at Bengaluru Tech Summit 2026, 17 to 19 November, BIEC Hall 1, booth H1A106. Book 20 minutes

Start For Free

Fail over between LLM providers

Try backup providers in your order, balance traffic across keys and remove failing keys with a circuit breaker, at the gateway.

Learning how to fail over between LLM providers usually means writing retry logic into every application. DeepintShield does it once, at the gateway: it tries backup providers and models in the order you set, spreads traffic across keys, and removes failing keys with a circuit breaker, all behind one OpenAI-compatible API.

The challenge

Every model provider has outages, rate limits and slow periods. Handling them in application code causes its own problems:

  • Different SDKs, different errors. Each provider signals failure in its own way, so retry code multiplies.
  • Rate limits hit at peak. HTTP 429 responses arrive exactly when traffic matters most.
  • Silent quality changes. A fallback model may answer differently, and nobody notices which model served a request.
  • Controls get skipped. Hand-written retries can bypass logging, budgets and guardrails.

The approach

Define fallback chains centrally and test them. Treat every attempt as a full request, so the same governance applies. Distinguish errors worth retrying from requests that were refused on purpose. Spread load across several keys and stop sending traffic to keys that are failing. Record which provider and key served each request so quality and cost can be checked afterwards. Keep chains short: every extra attempt adds latency and cost when a provider is struggling. Decide in advance which models are acceptable substitutes for each task, and which tasks should fail rather than fall back.

How to fail over between LLM providers with DeepintShield

  1. Define fallbacks. Send a fallbacks list in the request (in Python, through extra_body), or give a Virtual Key several providers so it builds a chain in weight order when the request has none. Fallbacks trigger on network errors, HTTP 500, 502, 503 and 504, 429, model unavailable, timeouts and authentication failures. They do not trigger on validation errors or guardrail and governance blocks. Each attempt is a fresh request, so caching, governance and logging apply again. See the fallbacks docs.
  2. Spread load across keys. Give each provider key a weight; the gateway uses weighted random selection and moves to the next eligible key if one fails. Enable the circuit breaker with load_balancer_enabled to remove failing keys and retry them periodically. On Team and above, adaptive load balancing shifts traffic by error rate, latency and recovery, with key states Healthy, Recovering and Circuit Open. See adaptive load balancing.
  3. Watch which provider served. The serving provider is returned in extra_fields.provider. Prometheus labels include the selected key and the fallback index. If every attempt fails, the original primary error is returned. Alert on the fallback rate as well as the error rate.

The gateway does not route automatically across providers for a model you did not configure, so list fallback models explicitly and check their answer quality for your task.

What you can show afterwards

  • Which provider and key served each request, and how often fallbacks ran.
  • Key health over time, including when keys entered Circuit Open and recovered.
  • Error rates by provider, ready for a capacity or vendor review.
  • Governance and logging records for fallback attempts, the same as for first attempts.

Run a failover drill before you need it. Force a timeout or errors on the primary key, confirm the fallback serves the request, check that the response names the serving provider, and find the fallback index in your metrics. Then compare a sample of fallback answers with primary answers for quality.

Planning a move to a gateway? Read why migration is more than a base URL change.

Start For Free