DEVSAIO

Model routing and fallbacks: use the right model every time

Send each request to the cheapest model that can handle it, and fall back automatically when a model is down.

DEVSAIO team · 2 min read
Share on X

With one API for every model, you can route intelligently instead of picking one model for everything.

Pattern 1: route by task type

The simplest router is a lookup in your code:

python
MODELS = {
    "classify": "claude-haiku-5-5",
    "chat": "claude-sonnet-4-6",
    "code": "claude-sonnet-4-6",
    "hard": "claude-opus-4-7",
}

Each feature in your app knows what it is, so it picks its model.

Pattern 2: route by difficulty

Let a cheap model judge difficulty first:

python
def pick(question):
    r = client.chat.completions.create(model="claude-haiku-5-5", temperature=0, max_tokens=5, messages=[
        {"role": "user", "content": f"Is this request simple or complex? Reply with one word.\n\n{question}"}])
    return "claude-opus-4-7" if "complex" in r.choices[0].message.content.lower() else "claude-sonnet-4-6"

The routing call is tiny, so it costs almost nothing.

Pattern 3: fall back on failure

python
import openai

CHAIN = ["claude-sonnet-4-6", "gpt-5.6-luna", "gemini-3.5-flash"]

def robust_chat(messages):
    last = None
    for model in CHAIN:
        try:
            return client.chat.completions.create(model=model, messages=messages, timeout=60)
        except (openai.APIConnectionError, openai.APITimeoutError, openai.RateLimitError, openai.InternalServerError) as e:
            last = e
            continue
    raise last

Only fall back on errors that are about availability (timeouts, 429, 5xx). A 400 means your request is wrong, and another model won't fix it.

Pattern 4: escalate on low confidence

Ask the first model to say ESCALATE when it isn't sure, then retry with a stronger model. This works well for support bots.

Measure

Log which model handled each request. Usage Logs show cost per model, so you can see whether the routing is actually saving money.

Try it with one API key

Claude, GPT, Gemini, DeepSeek, image and video models through one OpenAI- and Anthropic-compatible endpoint. Prepaid, pay only for what you use.

Keep reading