LLM Routing and Cascade Strategy: Cost Optimization with Multi-Model Orchestration

• Introduction
• What LLM Routing Is
  • Defining Routing and Why It Is Needed
  • Routing vs. Cascade vs. Ensemble
• Comparing the Main Routing Approaches
  • Detailed Comparison Table by Approach
  • RouteLLM: A Learned Router
  • FrugalGPT: Cascade-Based Cost Optimization
  • Semantic Router: Ultra-Fast Embedding-Based Routing
• Production Multi-Model Orchestration Architecture
  • Overall System Structure
  • TypeScript Orchestration Engine
  • Semantic Cache Integration
• Advanced Cost Optimization Strategies
  • Token Cost Analysis Framework
  • Prompt Compression and Token Savings
• Failure Patterns and Recovery Strategies
  • Main Failure Scenarios
  • Implementing the Circuit Breaker Pattern
• Troubleshooting Guide
  • Diagnosing Routing Quality Degradation
  • Analyzing a Cost Spike
  • Latency Optimization
• Operational Cautions
  • Managing the Cost-Quality Trade-off
  • Responding to Model Updates
  • Multi-Provider API Key Management
• Production Checklist
  • Pre-Adoption Checklist
  • Implementation Checklist
  • Operation Checklist
  • Performance Target Guidelines
• xRouter and the Future of Reinforcement-Learning Routing
• Guide by Real-World Adoption Scenario
  • Scenario 1: SaaS Chatbot Service (monthly cost target $5,000 → $1,500)
  • Scenario 2: Code Review Tool (quality > cost)
  • Scenario 3: RAG Pipeline (bulk document processing)
• References

Introduction

When you run an LLM-based service, sending every query to the highest-performing model (GPT-4o, Claude Opus and the like) looks like the safest choice. Reality is different. Between 60 and 80% of production traffic consists of simple queries such as "tell me today's weather" or "summarize this text", and using a top-tier model for those requests wastes money. Per token, GPT-4o costs roughly 30 times or more what GPT-4o-mini costs, and Claude Opus is roughly 60 times more expensive than Haiku.

The core strategies for solving this are LLM routing and cascading. Either you analyze query complexity, domain and required quality in real time and dispatch to the optimal model, or you try models in order from the cheapest and return immediately once the quality bar is met. According to the RouteLLM benchmark, a trained router can achieve an 85% cost reduction against GPT-4 while still holding 95% of the response quality.

This article compares the core concepts of LLM routing with the main approaches (RouteLLM, FrugalGPT, Semantic Router, Martian, Not Diamond), then builds, with code, a multi-model orchestration architecture you can apply in production right away. It also covers the failure patterns that surface in operation, recovery strategies, and a cost optimization checklist.

What LLM Routing Is

Defining Routing and Why It Is Needed

LLM routing is a decision layer that analyzes the user query and forwards the request to the most suitable model. Just as a network router forwards packets along the optimal path, an LLM router dispatches each query to the model with the best cost-quality-latency balance.

The fundamental reason routing is needed is the size of the cost-performance gap between models. Using GPT-4o for a simple classification task raises accuracy slightly but multiplies cost by tens of times. Conversely, using a small model for a task that needs complex reasoning degrades quality enough to force retries, so total cost can rise instead.

Routing vs. Cascade vs. Ensemble

The three approaches take different strategies.

Routing: analyze the query, select a single model, and call it once. Latency overhead is low and the implementation is relatively simple. The drawback is that when the router judges wrong, quality degrades immediately.

Cascade: call models in sequence starting from the cheapest, evaluate response quality, and escalate to a higher model when it falls short of the bar. This favors quality assurance, but average latency can increase.

Ensemble: call several models at once and combine the responses. Quality is the highest, but both cost and latency rise, so realistically it is used only in high-trust domains such as medicine and law.

In production, mixing routing and cascading is the most effective. The router performs the level-1 classification, and only when router confidence is low does the system fall back to a cascade.

Comparing the Main Routing Approaches

Detailed Comparison Table by Approach

RouteLLM: A Learned Router

RouteLLM is an open-source routing framework developed at LMSYS. Using preference data from Chatbot Arena, it trains a classifier that predicts "the probability that the strong model (GPT-4o) gives a better response to this query than the weak model (GPT-4o-mini)".

Four routers are provided. The MF (Matrix Factorization) router factorizes query embeddings and model characteristics into matrices to predict the win rate. The SW (Similarity-Weighted) router takes a weighted average of past win rates on similar queries. The BERT router learns routing directly with a BERT classifier. The Causal LLM router fine-tunes an LLM itself to make the routing decision.

# Running the RouteLLM server and calling it from a client
# pip install routellm

# 1. Start the server (OpenAI API compatible)
# python -m routellm.openai_server \
#   --routers mf \
#   --strong-model gpt-4o \
#   --weak-model gpt-4o-mini

# 2. Use it from the client
import openai

client = openai.OpenAI(
    base_url="http://localhost:6060/v1",
    api_key="not-needed",  # managed by the RouteLLM server
)

# The router selects the strong/weak model automatically
# The threshold value tunes routing sensitivity (0.0~1.0)
response = client.chat.completions.create(
    model="router-mf-0.11593",  # router-{router-name}-{threshold}
    messages=[
        {"role": "user", "content": "Explain how qubit error correction works in quantum computing"}
    ],
)

# The lower the threshold, the larger the share of strong-model calls
# 0.5 → ~50% strong model usage
# 0.1 → ~90% strong model usage (quality first)
# 0.9 → ~10% strong model usage (cost first)
print(response.choices[0].message.content)
print(f"Model used: {response.model}")

FrugalGPT: Cascade-Based Cost Optimization

FrugalGPT is an approach proposed at Stanford University that combines a model cascade with a quality estimator. The core idea is to try the cheapest model first and return immediately once response quality is good enough.

The flow works as follows. When a query arrives it goes first to the cheapest model (for example GPT-4o-mini). The quality estimator scores the response and returns it when the score is at or above the threshold. If it falls short, the same query goes to the next model up (for example Claude Sonnet) and quality judgment runs again. Once it reaches the top model (for example GPT-4o), the response is returned unconditionally.

# FrugalGPT-style cascade implementation
from openai import OpenAI
from anthropic import Anthropic
import time
from dataclasses import dataclass
from typing import Optional

@dataclass
class ModelTier:
    name: str
    provider: str
    cost_per_1k_tokens: float
    quality_threshold: float  # return the response at or above this score

# Model tiers listed in ascending order of cost
MODEL_CASCADE = [
    ModelTier("gpt-4o-mini", "openai", 0.00015, 0.7),
    ModelTier("claude-3-5-haiku-20241022", "anthropic", 0.001, 0.8),
    ModelTier("claude-sonnet-4-20250514", "anthropic", 0.003, 0.85),
    ModelTier("gpt-4o", "openai", 0.005, 0.0),  # final stage: always returns
]

openai_client = OpenAI()
anthropic_client = Anthropic()


def call_model(model: ModelTier, query: str) -> str:
    """Call the right API for the model provider."""
    if model.provider == "openai":
        resp = openai_client.chat.completions.create(
            model=model.name,
            messages=[{"role": "user", "content": query}],
            temperature=0.3,
        )
        return resp.choices[0].message.content
    elif model.provider == "anthropic":
        resp = anthropic_client.messages.create(
            model=model.name,
            max_tokens=2048,
            messages=[{"role": "user", "content": query}],
        )
        return resp.content[0].text


def estimate_quality(query: str, response: str) -> float:
    """Quality estimator - returns a quality score from a lightweight model."""
    judge_prompt = f"""Rate the quality of the question and answer below on a 0.0~1.0 scale.
Criteria: accuracy, completeness, relevance, clarity
Question: {query}
Answer: {response}
Return the number only (for example: 0.85)"""

    resp = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0.0,
        max_tokens=10,
    )
    try:
        return float(resp.choices[0].message.content.strip())
    except ValueError:
        return 0.0  # low score on a parse failure → escalate to the higher model


def frugal_cascade(query: str) -> dict:
    """Run the FrugalGPT cascade."""
    results = []

    for tier in MODEL_CASCADE:
        start = time.time()
        response = call_model(tier, query)
        latency = time.time() - start

        # The final tier skips quality judgment
        if tier.quality_threshold == 0.0:
            return {
                "response": response,
                "model": tier.name,
                "latency": latency,
                "cascade_depth": len(results) + 1,
                "total_cost_ratio": sum(r["cost"] for r in results) + tier.cost_per_1k_tokens,
            }

        quality = estimate_quality(query, response)

        results.append({
            "model": tier.name,
            "quality": quality,
            "cost": tier.cost_per_1k_tokens,
            "latency": latency,
        })

        if quality >= tier.quality_threshold:
            return {
                "response": response,
                "model": tier.name,
                "quality_score": quality,
                "latency": latency,
                "cascade_depth": len(results),
            }

    # Theoretically unreachable (the last tier always returns)
    return {"response": response, "model": MODEL_CASCADE[-1].name}

Semantic Router: Ultra-Fast Embedding-Based Routing

Semantic Router is a library developed by Aurelio Labs that makes the routing decision from the semantic similarity of the query. Because it decides purely from cosine similarity between embedding vectors without calling an LLM, latency overhead is extremely low, on the order of 2~5ms.

# Per-domain model routing with Semantic Router
# pip install semantic-router

from semantic_router import Route, RouteLayer
from semantic_router.encoders import OpenAIEncoder

# Route definition: give each route a set of representative utterances
simple_route = Route(
    name="simple",
    utterances=[
        "What is the weather today?",
        "What is the population of Seoul?",
        "Hello",
        "What does this word mean?",
        "What is 1+1?",
        "Tell me the current time",
    ],
)

coding_route = Route(
    name="coding",
    utterances=[
        "Implement quicksort in Python",
        "Fix a useEffect memory leak in a React component",
        "Debug an OOMKilled Kubernetes Pod",
        "How to optimize a SQL query",
        "Code comparing gRPC and REST API performance",
    ],
)

reasoning_route = Route(
    name="reasoning",
    utterances=[
        "Critically analyze the methodology of this paper",
        "Explain the correlation between GDP growth and unemployment in economic terms",
        "Derive the difference between quantum entanglement and quantum teleportation mathematically",
        "Compare and analyze the theoretical limits of RLHF and DPO",
    ],
)

# Initialize the encoder and the route layer
encoder = OpenAIEncoder(name="text-embedding-3-small")
route_layer = RouteLayer(
    encoder=encoder,
    routes=[simple_route, coding_route, reasoning_route],
)

# Model mapping per route
MODEL_MAP = {
    "simple": "gpt-4o-mini",        # low-cost model
    "coding": "claude-sonnet-4-20250514",  # coding-specialized
    "reasoning": "gpt-4o",          # high-performance reasoning
    None: "claude-sonnet-4-20250514",      # default fallback
}


def route_query(query: str) -> dict:
    """Analyze the query and pick the optimal model."""
    route_result = route_layer(query)
    selected_model = MODEL_MAP.get(route_result.name, MODEL_MAP[None])

    return {
        "query": query,
        "route": route_result.name,
        "confidence": route_result.similarity_score,
        "model": selected_model,
    }


# Usage example
queries = [
    "How to sort a list in Python",
    "Explain the attention mechanism of the transformer architecture mathematically",
    "Busan weather tomorrow",
]

for q in queries:
    result = route_query(q)
    print(f"Query: {q}")
    print(f"  Route: {result['route']} → Model: {result['model']}")
    print(f"  Confidence: {result['confidence']:.3f}")

Production Multi-Model Orchestration Architecture

Overall System Structure

When you build multi-model orchestration in a production environment, you have to integrate more than plain routing: observability, fallback, caching and rate limiting. Below is a production-grade design for the orchestration layer.

User request → API Gateway → semantic cache lookup
                              ↓ (cache miss)
                         Query classifier (complexity/domain analysis)
                              ↓
                    ┌─────────┼─────────┐
                    ↓         ↓         ↓
                  Small      Medium       Large
              (GPT-4o-mini) (Sonnet)  (GPT-4o/Opus)
                    ↓         ↓         ↓
                    └─────────┼─────────┘
                              ↓
                         Quality gate (cascade decision)
                              ↓
                    ┌── Quality met   → return response + store in cache
                    └── Quality short → escalate to the higher model
                              ↓
                         Metrics collection (cost/latency/quality)

TypeScript Orchestration Engine

// multi-model-orchestrator.ts
import OpenAI from 'openai'
import Anthropic from '@anthropic-ai/sdk'

interface ModelConfig {
  id: string
  provider: 'openai' | 'anthropic'
  costPer1kInput: number
  costPer1kOutput: number
  maxTokens: number
  tier: 'small' | 'medium' | 'large'
}

interface RoutingDecision {
  model: ModelConfig
  reason: string
  confidence: number
}

interface OrchestratorResult {
  response: string
  model: string
  tier: string
  latencyMs: number
  estimatedCost: number
  cascadeDepth: number
  cacheHit: boolean
}

// Model catalog definition
const MODEL_CATALOG: ModelConfig[] = [
  {
    id: 'gpt-4o-mini',
    provider: 'openai',
    costPer1kInput: 0.00015,
    costPer1kOutput: 0.0006,
    maxTokens: 16384,
    tier: 'small',
  },
  {
    id: 'claude-3-5-haiku-20241022',
    provider: 'anthropic',
    costPer1kInput: 0.001,
    costPer1kOutput: 0.005,
    maxTokens: 8192,
    tier: 'small',
  },
  {
    id: 'claude-sonnet-4-20250514',
    provider: 'anthropic',
    costPer1kInput: 0.003,
    costPer1kOutput: 0.015,
    maxTokens: 8192,
    tier: 'medium',
  },
  {
    id: 'gpt-4o',
    provider: 'openai',
    costPer1kInput: 0.005,
    costPer1kOutput: 0.015,
    maxTokens: 16384,
    tier: 'large',
  },
]

class QueryClassifier {
  private openai: OpenAI

  constructor(openai: OpenAI) {
    this.openai = openai
  }

  async classify(query: string): Promise<RoutingDecision> {
    // Rule-based level-1 classification (decided immediately, no LLM call)
    const ruleResult = this.ruleBasedClassify(query)
    if (ruleResult) return ruleResult

    // Level-2 classification with a lightweight model
    const resp = await this.openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [
        {
          role: 'system',
          content: `Classify the complexity of the query.
Respond in JSON: {"tier": "small|medium|large", "reason": "...", "confidence": 0.0~1.0}
- small: simple questions, greetings, translation, summarization
- medium: coding, analysis, comparison, structured explanation
- large: compound reasoning, mathematical proof, multi-step analysis, creative long-form`,
        },
        { role: 'user', content: query },
      ],
      response_format: { type: 'json_object' },
      temperature: 0.0,
      max_tokens: 100,
    })

    const parsed = JSON.parse(resp.choices[0].message.content || '{}')
    const tier = parsed.tier || 'medium'
    const model = MODEL_CATALOG.find((m) => m.tier === tier) || MODEL_CATALOG[2]

    return {
      model,
      reason: parsed.reason || 'LLM classifier decision',
      confidence: parsed.confidence || 0.5,
    }
  }

  private ruleBasedClassify(query: string): RoutingDecision | null {
    const len = query.length

    // Very short query → small
    if (len < 30) {
      return {
        model: MODEL_CATALOG[0],
        reason: 'Short query (rule-based)',
        confidence: 0.9,
      }
    }

    // Code-related keywords → medium
    const codeKeywords = /\b(code|implement|function|class|debug|error|API|SQL|React|Python)\b/i
    if (codeKeywords.test(query)) {
      return {
        model: MODEL_CATALOG[2], // claude-sonnet
        reason: 'Coding-related query (rule-based)',
        confidence: 0.8,
      }
    }

    // Complex reasoning keywords → large
    const reasoningKeywords = /\b(proof|analysis|compare.*difference|mathematical|logical|strategy.*establish|architecture.*design)\b/
    if (reasoningKeywords.test(query)) {
      return {
        model: MODEL_CATALOG[3], // gpt-4o
        reason: 'Compound reasoning query (rule-based)',
        confidence: 0.75,
      }
    }

    return null // Rules cannot decide → use the LLM classifier
  }
}

Semantic Cache Integration

Reusing a previous response for an identical or similar query cuts cost dramatically. With embedding-based similarity search rather than exact string matching, "how to sort a Python list" and "how do I sort a list in Python" can be recognized as the same query.

# Semantic cache implementation (Redis + vector similarity)
import hashlib
import json
import time
import numpy as np
from openai import OpenAI
from redis import Redis

client = OpenAI()
redis_client = Redis(host="localhost", port=6379, db=0)

CACHE_TTL = 3600  # 1 hour
SIMILARITY_THRESHOLD = 0.92  # similarity threshold


def get_embedding(text: str) -> list[float]:
    """Create the embedding vector for the text."""
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return resp.data[0].embedding


def cosine_similarity(a: list[float], b: list[float]) -> float:
    """Compute cosine similarity."""
    a_np, b_np = np.array(a), np.array(b)
    return float(np.dot(a_np, b_np) / (np.linalg.norm(a_np) * np.linalg.norm(b_np)))


class SemanticCache:
    def __init__(self, namespace: str = "llm_cache"):
        self.namespace = namespace

    def _cache_key(self, idx: int) -> str:
        return f"{self.namespace}:entry:{idx}"

    def _counter_key(self) -> str:
        return f"{self.namespace}:counter"

    def get(self, query: str) -> dict | None:
        """Look up a cached response for a similar query."""
        query_embedding = get_embedding(query)
        counter = int(redis_client.get(self._counter_key()) or 0)

        best_match = None
        best_similarity = 0.0

        for i in range(counter):
            entry_raw = redis_client.get(self._cache_key(i))
            if not entry_raw:
                continue

            entry = json.loads(entry_raw)
            similarity = cosine_similarity(query_embedding, entry["embedding"])

            if similarity > best_similarity and similarity >= SIMILARITY_THRESHOLD:
                best_similarity = similarity
                best_match = entry

        if best_match:
            return {
                "response": best_match["response"],
                "model": best_match["model"],
                "similarity": best_similarity,
                "cached_at": best_match["timestamp"],
            }

        return None

    def put(self, query: str, response: str, model: str):
        """Store the response in the cache."""
        embedding = get_embedding(query)
        counter = int(redis_client.get(self._counter_key()) or 0)

        entry = {
            "query": query,
            "response": response,
            "model": model,
            "embedding": embedding,
            "timestamp": time.time(),
        }

        redis_client.setex(
            self._cache_key(counter),
            CACHE_TTL,
            json.dumps(entry),
        )
        redis_client.incr(self._counter_key())

Advanced Cost Optimization Strategies

Token Cost Analysis Framework

The first step in cost optimization is to understand the current cost structure precisely. Track cost by model, by feature and by time of day so you can identify the areas with the most room to optimize.

# Cost tracking and analysis system
from dataclasses import dataclass, field
from collections import defaultdict
from datetime import datetime, timedelta
import json


@dataclass
class TokenUsage:
    model: str
    input_tokens: int
    output_tokens: int
    timestamp: datetime
    route: str  # which route it was classified into
    cascade_depth: int = 1
    quality_score: float = 0.0


class CostAnalyzer:
    # Prices for the main models as of March 2026 (USD per 1K tokens)
    PRICING = {
        "gpt-4o": {"input": 0.0025, "output": 0.01},
        "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
        "claude-opus-4-20250514": {"input": 0.015, "output": 0.075},
        "claude-sonnet-4-20250514": {"input": 0.003, "output": 0.015},
        "claude-3-5-haiku-20241022": {"input": 0.001, "output": 0.005},
    }

    def __init__(self):
        self.usage_log: list[TokenUsage] = []

    def log(self, usage: TokenUsage):
        self.usage_log.append(usage)

    def calculate_cost(self, usage: TokenUsage) -> float:
        """Cost of a single call."""
        pricing = self.PRICING.get(usage.model)
        if not pricing:
            return 0.0

        input_cost = (usage.input_tokens / 1000) * pricing["input"]
        output_cost = (usage.output_tokens / 1000) * pricing["output"]
        return input_cost + output_cost

    def daily_report(self, date: datetime = None) -> dict:
        """Build the daily cost report."""
        date = date or datetime.now()
        day_start = date.replace(hour=0, minute=0, second=0)
        day_end = day_start + timedelta(days=1)

        day_logs = [
            u for u in self.usage_log
            if day_start <= u.timestamp < day_end
        ]

        model_costs = defaultdict(float)
        route_costs = defaultdict(float)
        total_cost = 0.0
        total_requests = len(day_logs)

        for usage in day_logs:
            cost = self.calculate_cost(usage)
            model_costs[usage.model] += cost
            route_costs[usage.route] += cost
            total_cost += cost

        # Estimated cost if every request had gone to GPT-4o with no routing
        baseline_cost = sum(
            (u.input_tokens / 1000) * self.PRICING["gpt-4o"]["input"]
            + (u.output_tokens / 1000) * self.PRICING["gpt-4o"]["output"]
            for u in day_logs
        )

        return {
            "date": date.strftime("%Y-%m-%d"),
            "total_requests": total_requests,
            "total_cost_usd": round(total_cost, 4),
            "baseline_cost_usd": round(baseline_cost, 4),
            "savings_pct": round((1 - total_cost / baseline_cost) * 100, 1) if baseline_cost > 0 else 0,
            "cost_by_model": dict(model_costs),
            "cost_by_route": dict(route_costs),
            "avg_cost_per_request": round(total_cost / total_requests, 6) if total_requests > 0 else 0,
        }

Prompt Compression and Token Savings

Beyond routing, you can cut token usage by optimizing the prompt itself. Compressing the system prompt, removing unnecessary context and capping output tokens can cut cost by 20~40% even on the same model.

The main techniques are as follows. System prompt caching: using the Prompt Caching feature from Anthropic cuts the cost of a repeated system prompt by 90%. LLMLingua prompt compression: removes low-importance tokens from the original prompt for 2~5x compression. Output length control: cap max_tokens to fit the task and write prompts that elicit concise responses.

Failure Patterns and Recovery Strategies

Main Failure Scenarios

1. Router misclassification (misrouting)

When the router misclassifies a complex query as a simple one and sends it to a small model, quality drops sharply. Conversely, judging a simple query to be complex incurs unnecessary cost.

Recovery strategy: build a user feedback loop to collect misclassification cases, and retrain the router periodically. For low-confidence classifications (confidence < 0.6), add a safeguard that automatically selects the middle-tier model.

2. Cascade latency blowup

In a cascade, if the cheap models fail the quality bar one after another, the request walks every tier and latency spikes. In a 4-stage cascade where each stage takes 1~2 seconds, the worst case produces a response time of 8 seconds or more.

Recovery strategy: cap the maximum cascade depth (usually 2~3 stages). Set an overall timeout and, when it fires, return the best response so far. Monitor the distribution of cascade depth and recalibrate the router when the average depth exceeds 1.5.

3. Provider outage

When a particular model provider (OpenAI, Anthropic, Google and so on) has an outage, every routing path that uses that provider stops.

Recovery strategy: implement a per-provider health check and, on detecting an outage, fall back automatically to an alternative model in the same tier. Apply the circuit breaker pattern so a provider is temporarily cut off after consecutive failures.

4. Quality estimator drift

This is the phenomenon where the FrugalGPT quality estimator grows inaccurate over time. Model updates, shifts in the data distribution and bias in the estimator itself are the causes.

Recovery strategy: periodically compare human evaluator labels against estimator scores to detect drift. Retrain the estimator model periodically as well, or compare estimator versions with an A/B test.

Implementing the Circuit Breaker Pattern

# Per-provider circuit breaker implementation
import time
from enum import Enum
from threading import Lock


class CircuitState(Enum):
    CLOSED = "closed"      # normal state
    OPEN = "open"          # blocked state
    HALF_OPEN = "half_open"  # trial state


class CircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,
        recovery_timeout: int = 60,
        half_open_max_calls: int = 3,
    ):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.half_open_max_calls = half_open_max_calls

        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.success_count = 0
        self.last_failure_time = 0.0
        self.half_open_calls = 0
        self._lock = Lock()

    def can_execute(self) -> bool:
        """Whether a request may run in the current state."""
        with self._lock:
            if self.state == CircuitState.CLOSED:
                return True

            if self.state == CircuitState.OPEN:
                # Move to HALF_OPEN once the recovery timeout has elapsed
                if time.time() - self.last_failure_time >= self.recovery_timeout:
                    self.state = CircuitState.HALF_OPEN
                    self.half_open_calls = 0
                    return True
                return False

            if self.state == CircuitState.HALF_OPEN:
                return self.half_open_calls < self.half_open_max_calls

            return False

    def record_success(self):
        """Record a success."""
        with self._lock:
            if self.state == CircuitState.HALF_OPEN:
                self.success_count += 1
                if self.success_count >= self.half_open_max_calls:
                    self.state = CircuitState.CLOSED
                    self.failure_count = 0
                    self.success_count = 0
            elif self.state == CircuitState.CLOSED:
                self.failure_count = 0

    def record_failure(self):
        """Record a failure."""
        with self._lock:
            self.failure_count += 1
            self.last_failure_time = time.time()

            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.OPEN  # block again
            elif self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN


# Per-provider circuit breaker registry
provider_circuits: dict[str, CircuitBreaker] = {
    "openai": CircuitBreaker(failure_threshold=5, recovery_timeout=60),
    "anthropic": CircuitBreaker(failure_threshold=5, recovery_timeout=60),
    "google": CircuitBreaker(failure_threshold=3, recovery_timeout=120),
}

# Same-tier fallback mapping
FALLBACK_MAP = {
    "gpt-4o-mini": ["claude-3-5-haiku-20241022"],
    "claude-3-5-haiku-20241022": ["gpt-4o-mini"],
    "claude-sonnet-4-20250514": ["gpt-4o"],
    "gpt-4o": ["claude-sonnet-4-20250514"],
}


def get_available_model(
    primary_model: str,
    primary_provider: str,
) -> tuple[str, str]:
    """Return an available model (circuit breaker check included)."""
    # Level 1: check the provider of the primary model
    if provider_circuits[primary_provider].can_execute():
        return primary_model, primary_provider

    # Level 2: search the fallback models
    fallbacks = FALLBACK_MAP.get(primary_model, [])
    for fb_model in fallbacks:
        fb_provider = "anthropic" if "claude" in fb_model else "openai"
        if provider_circuits[fb_provider].can_execute():
            return fb_model, fb_provider

    # Level 3: error when every provider is down
    raise RuntimeError(
        f"All provider circuits are OPEN. "
        f"primary={primary_model}, fallbacks={fallbacks}"
    )

Troubleshooting Guide

Diagnosing Routing Quality Degradation

Symptoms: rising user dissatisfaction, a higher retry rate, quality collapsing on particular query types

The diagnostic procedure is as follows.

1. Check the routing distribution: verify that the traffic share of each model tier sits in the expected range. If the small-model share exceeds 80%, the router is likely over-optimized for cost.

2. Measure the misclassification rate: have human evaluators reclassify 100~500 randomly sampled queries and compare against the router classification. A misclassification rate above 15% means the router needs retraining.

3. Compare quality scores by tier: compare the average quality score of the queries handled at each tier. If the small-model tier averages below 0.7, the threshold needs adjusting.

4. Analyze cascade depth: an average cascade depth above 1.5 means the level-1 routing is inaccurate.

Analyzing a Cost Spike

Symptoms: daily cost suddenly increases 2x or more

Here is what to check.

1. Traffic spike: check whether total request volume went up.
2. Routing distribution shift: check whether the large-model share spiked. A router update or a change in the query distribution can be the cause.
3. Cache hit rate drop: check for cache TTL expiry, a cache server outage, or a new type of query arriving.
4. Cascade loop: check whether a malfunctioning quality estimator is escalating every query all the way to the top model.
5. Prompt bloat: check whether the system prompt or the context has grown abnormally large.

Latency Optimization

Minimizing the latency of the routing layer itself matters. If the router takes 50ms or more, it affects the performance users feel.

The optimization methods are as follows. Run rule-based classification at level 1 so that 70% of queries route immediately (latency < 1ms). Send only the remaining 30% through embedding-based or LLM-based classification. Deploy the classifier model locally where possible to remove network latency. Batch the embedding computations of Semantic Router to raise throughput.

Operational Cautions

Managing the Cost-Quality Trade-off

Adopting a routing strategy creates continuous tension between cost and quality. Cut cost aggressively and quality falls; raise quality and the cost savings from routing shrink.

The operating principles are as follows.

• Set thresholds from the SLA: set the routing threshold against a quality SLA (for example, 95%ile satisfaction of 4.0 or higher) rather than a cost reduction target. Cost savings are a result, not the goal.
• Gradual rollout: apply a new routing policy in stages, 5% → 20% → 50% → 100% of total traffic. Check the quality metrics at each stage.
• Automatic rollback criteria: roll back to the previous routing policy automatically when the quality score falls 10% or more below the baseline.

Responding to Model Updates

When an LLM provider updates a model (GPT-4o → GPT-4o-2024-11-20 and so on), the training data of the router and the current model behavior can diverge. Follow this procedure when a model is updated.

1. Run benchmarks for the new model version in a staging environment.
2. Update the model profile of the router (cost, performance characteristics).
3. Compare the existing routing against routing based on the new model with an A/B test.
4. Retrain the router if there is a meaningful difference.

Multi-Provider API Key Management

Using several providers increases the complexity of API key management. Environment variables, a secret manager and key rotation all need systematic handling.

// Secure multi-provider API key management
// provider-config.ts
import { SecretManagerServiceClient } from '@google-cloud/secret-manager'

interface ProviderCredentials {
  apiKey: string
  orgId?: string
  rateLimit: number // RPM
  lastRotated: Date
}

class ProviderKeyManager {
  private secretClient: SecretManagerServiceClient
  private cache: Map<string, ProviderCredentials> = new Map()
  private cacheTTL = 300_000 // 5 minutes
  private lastFetch: Map<string, number> = new Map()

  constructor() {
    this.secretClient = new SecretManagerServiceClient()
  }

  async getCredentials(provider: string): Promise<ProviderCredentials> {
    const now = Date.now()
    const lastFetched = this.lastFetch.get(provider) || 0

    // Return from the cache while the cache is still valid
    if (now - lastFetched < this.cacheTTL && this.cache.has(provider)) {
      return this.cache.get(provider)!
    }

    // Fetch the key from Secret Manager
    const secretName = `projects/my-project/secrets/llm-${provider}-key/versions/latest`
    const [version] = await this.secretClient.accessSecretVersion({
      name: secretName,
    })

    const apiKey = version.payload?.data?.toString() || ''
    const credentials: ProviderCredentials = {
      apiKey,
      rateLimit: this.getDefaultRateLimit(provider),
      lastRotated: new Date(),
    }

    this.cache.set(provider, credentials)
    this.lastFetch.set(provider, now)

    return credentials
  }

  private getDefaultRateLimit(provider: string): number {
    const limits: Record<string, number> = {
      openai: 500,
      anthropic: 300,
      google: 200,
    }
    return limits[provider] || 100
  }
}

Production Checklist

Pre-Adoption Checklist

• Current LLM cost structure analyzed (by model, by feature, by time of day)
• Query complexity distribution analyzed (simple/medium/complex ratio)
• Quality SLA defined (minimum quality score, maximum latency)
• Routing strategy chosen (rule-based / learned / cascade / hybrid)
• Candidate model catalog assembled (at least 3 models, spread across price and performance)
• Benchmark dataset built (100+ queries per domain, expected quality labels)
• Cost reduction target set (target reduction against the baseline)

Implementation Checklist

• Router module implemented and unit tested
• Per-provider circuit breaker applied
• Semantic cache built and TTL configured
• Cost tracking metric collection pipeline built
• Quality estimator implemented and calibrated
• Fallback chain configured (alternative path on a provider outage)
• Rate limit management (per-provider RPM/TPM limits respected)
• Logging: routing decisions, model responses, cost and latency recorded

Operation Checklist

• Daily cost report dashboard built
• Routing distribution monitored (traffic share by tier)
• Quality score trend monitored (7-day moving average)
• Cache hit rate monitored (target: 20~40%)
• Cascade depth distribution monitored (average < 1.5)
• Circuit breaker state alerting configured
• Weekly misclassification sample review (50~100 cases)
• Monthly router retraining / threshold recalibration
• Benchmark rerun process for model updates
• Automatic cost spike alert (50% or more increase over the previous day)

Performance Target Guidelines

xRouter and the Future of Reinforcement-Learning Routing

The approach worth watching in recent research is xRouter. xRouter formalizes routing as a sequential decision-making problem and trains the router with reinforcement learning (RL). Where existing methods learn a static mapping between queries and models, xRouter learns a policy that maximizes the cumulative reward across a whole session under a cost budget constraint.

The key to this approach is that it models cost as an explicit constraint. Because it directly optimizes the goal "maximize quality under a total cost of $X or less", it adjusts the cost-quality trade-off automatically within the budget. When the cost budget is generous it makes heavy use of large models, and as the budget is consumed it shows adaptive behavior, reaching for small models more often.

The Pick and Spin framework is also worth watching. Combining adaptive scale-to-zero automation with a hybrid routing module in a Kubernetes-based self-hosted LLM environment, it achieved a 21.6% higher success rate, 30% lower latency and 33% lower GPU cost than static deployment.

In agentic AI workflows, routing matters even more. Because errors accumulate at every step when an agent runs several steps in sequence, routing that picks the optimal model per step has a decisive effect on the overall success rate. The Expert Orchestration AI Architecture from Martian proposes a structure in which "judge" models evaluate the abilities of "expert" models and the router assigns the query to the expert it can trust most.

Guide by Real-World Adoption Scenario

Scenario 1: SaaS Chatbot Service (monthly cost target $5,000 → $1,500)

This is the most common adoption scenario. In a customer support chatbot, 70% of all queries are often FAQ-level simple questions.

The recommended strategy is as follows. Define 3 routes with Semantic Router: FAQ, general and specialist. Route FAQ to cache + GPT-4o-mini, general to Claude Haiku, and specialist to Claude Sonnet. Apply a semantic cache and target a cache hit rate of 50% or more on repeated FAQ queries. This configuration generally achieves a 60~70% cost reduction.

Scenario 2: Code Review Tool (quality > cost)

Code analysis lives on accuracy, so the focus falls on guaranteeing quality rather than aggressive cost cutting.

The recommended strategy is as follows. Route by rule-based classification on the number of changed lines and the file type. Send simple changes under 10 lines to GPT-4o-mini, and compound changes of 50 lines or more or security-related files to GPT-4o or Claude Opus. Cap the cascade at 2 stages, and escalate to a higher model when the response from the first model carries uncertainty markers such as "not confident" or "needs further review".

Scenario 3: RAG Pipeline (bulk document processing)

In a RAG system, the model performance you need varies with the number of retrieved document chunks and the query complexity.

The recommended strategy is as follows. When retrieval returns 1~2 chunks and a direct answer is possible, handle it with GPT-4o-mini. When 5 or more chunks have to be synthesized, or comparison and analysis is needed, use a medium model. When conflicts between multiple documents have to be resolved or reasoning is required, use a large model. Always handle document summarization and embedding generation with a small model.

References

• RouteLLM: Cost-Effective LLM Routing (LMSYS) - The official RouteLLM blog post
• RouteLLM GitHub Repository - RouteLLM open-source code and documentation
• FrugalGPT: Smarter LLM Usage for Lower Costs (Portkey) - A FrugalGPT implementation guide
• Semantic Router GitHub (Aurelio Labs) - The Semantic Router open-source library
• Martian Model Router - The official Martian site
• Not Diamond - AI Model Router - The official Not Diamond site
• xRouter: Cost-Aware LLM Orchestration via RL (arXiv) - The reinforcement-learning router paper
• Efficient Multi-Model Orchestration (arXiv) - The Pick and Spin framework paper
• Multi-LLM Routing Strategies on AWS - A guide to multi-model routing on AWS
• LLM Routing - Intuitively and Exhaustively Explained (Towards Data Science) - An explainer on LLM routing concepts
• Dynamic Model Routing and Cascading Survey (arXiv) - A routing/cascade survey paper
• Awesome AI Model Routing (Not Diamond) - A curated list of AI model routing approaches