How we route between models

Every production agent we ship has a routing layer in front of the model calls. Not as a future-proofing exercise, and not out of loyalty to any one provider. It is there because, outside of the eval suite itself, no piece of infrastructure in a 2026 AI build moves cost and quality further for the effort.

The single decision of “which model handles this query” affects cost by a factor of 10 to 100, latency by 5 to 20, and quality by enough to matter only on the queries that need quality the most. Done well, the user sees fast cheap answers on the 85% of traffic where they are sufficient, and a slower better answer on the 15% where they are not. Done badly, the user sees inconsistent quality and the bill triples.

The shape of the router

The router we ship is small, deterministic, and observable. It is not a model. It is a function from (request, context) to (provider, model, parameters, fallback_chain). That function is testable in isolation, has its own unit evals, and is the first thing we add to an agent.

// src/router/index.ts
export interface RouteRequest {
  intent: string;          // tagged by an upstream classifier
  expected_tokens: number; // estimated by a heuristic over the prompt
  latency_budget_ms: number;
  cost_ceiling_cents: number;
  tenant_id: string;
  trace_id: string;
}

export interface RouteDecision {
  primary: ModelEndpoint;
  fallbacks: ModelEndpoint[];
  reason: string;          // structured, logged with every call
}

export function route(req: RouteRequest): RouteDecision {
  // deterministic rules first
  if (req.intent === 'compliance_review') {
    return rule('frontier_only', req);
  }
  if (req.expected_tokens > 8_000) {
    return rule('long_context', req);
  }
  if (req.intent === 'classification' || req.intent === 'extraction') {
    return rule('small_fast', req);
  }
  return rule('default_balanced', req);
}

The router is intentionally not “smart”. A rules table you can read in five minutes is easier to reason about, easier to roll back, and easier to defend to compliance than a learned routing policy that nobody on the team can explain when it routes wrong.

What we route on

Four axes, in priority order.

Intent. What the agent is being asked to do. Some intents have a non-negotiable model choice. A compliance review query routes to a frontier model regardless of cost. A high-volume classification routes to a small fast model regardless of latency budget. The intent classifier is itself a small evaled model, with a fallback to a deterministic keyword rule if the classifier is unavailable.

Expected context length. Long context is expensive everywhere and worth it only where it helps. We route requests with estimated input above 8k tokens to a long-context-capable endpoint. Requests below that are routed to whichever endpoint best matches the other constraints, because long-context endpoints are uniformly slower and pricier than their short-context siblings even when underused.

Latency budget. Synchronous user-facing flows have tight budgets. Batch and async flows do not. The router never picks a model whose 95th percentile latency exceeds the budget, even if it is the highest-quality option. We measure 95th not mean because the mean is a lie about what users experience.

Cost ceiling. Every request carries a maximum cost in cents. If the cheapest acceptable model exceeds the ceiling, the request fails fast with a structured error the caller can handle. We do not silently route to a worse model to fit the budget. Silent degradation is the worst possible failure mode in a regulated environment.

Provider mix

For most engagements the router is choosing between three categories of endpoint:

Tier	Examples	Use
Frontier	Anthropic Claude (top tier), OpenAI top tier	High-stakes intents, complex reasoning, compliance-sensitive work
Balanced	Anthropic Claude (mid tier), OpenAI mid tier	Default tier for most user-facing work
Small/fast	Anthropic Claude (small tier), OpenAI small tier, locally-hosted small open-weights	Classification, extraction, routing-itself, high-volume batch

We are deliberately model-agnostic in this document. Naming specific model versions in a doc that lives for years is a fast way to make the doc wrong. The router config in any given client repo names specific versions, with an evaluated_at date and a link to the eval run that approved the version.

Deterministic fallbacks

Every primary route comes with a fallback chain. The chain is deterministic and ordered: if the primary endpoint fails (5xx, timeout, rate limit, content filter trip), the router moves to the next in the chain without re-running the routing logic. Re-routing on failure introduces non-determinism that ruins reproducibility in incident review.

// example route table
const ROUTES: Record<RuleName, RouteDecision> = {
  frontier_only: {
    primary:   { provider: 'anthropic', model: 'claude-frontier' },
    fallbacks: [
      { provider: 'openai',    model: 'gpt-frontier' },
      { provider: 'anthropic', model: 'claude-balanced' }, // last-resort degrade, alerts on use
    ],
    reason: 'compliance_review:frontier_required',
  },
  default_balanced: {
    primary:   { provider: 'anthropic', model: 'claude-balanced' },
    fallbacks: [
      { provider: 'openai',    model: 'gpt-balanced' },
      { provider: 'local',     model: 'small-open-weights' }, // graceful degrade
    ],
    reason: 'default:balanced',
  },
  // ...
};

When a fallback fires, it is logged as a first-class event. A spike in route.fallback_used is an incident signal, not a “we handled it” signal. The system kept working, but the cost, latency, and quality assumptions the client signed off on are now being violated until the primary recovers.

Evaluating model upgrades

The hardest thing about a multi-provider router is upgrading any one model in it safely. The discipline we apply is the same we apply to every other change: it goes through the eval suite first.

For a model upgrade specifically:

1. Add the candidate model as a non-primary entry in a shadow route. The shadow runs every request the candidate would have handled on the primary, in parallel, and logs the candidate’s response without serving it.
2. Run the shadow for at least two weeks against real production traffic.
3. Score the shadow’s responses against the same regression rubric used for the live system, on a representative sample.
4. If the candidate scores within 1 point of the primary on overall pass rate, within 0 points on compliance_sensitive, and within budget on cost and latency, promote it.
5. If it doesn’t, document the gap and either upgrade with a narrowed scope (e.g. promote only for intents where it wins) or drop the candidate.

This sounds slow. It is slower than swapping a model name in a config file, and it is many orders of magnitude faster than recovering from a silent quality regression that shipped to production and broke trust with a stakeholder who is now in the room.

Cost ceilings, observably

The cost ceiling on every request is enforced by the router. The total cost of every call is recorded against tenant_id, intent, and model. We provide a dashboard that shows:

• Total spend per tenant per day.
• Spend by intent (so the client can see which agent behaviours dominate the bill).
• Spend by model (so the client can see what the routing decisions translate into).
• Spend per successful resolution (so the client can see unit economics, not just gross spend).

When a tenant approaches a budget cap, the router downgrades non-critical intents to small/fast models and alerts the operator. The system keeps working. The behaviour changes in a documented, configurable way. The operator is in the loop before the bill arrives.

What we do not do

We do not use a model to decide routing. We have looked at it. Learned routing policies are harder to debug, harder to defend to compliance, and not measurably better than a curated rules table on the eval set. The rules table also has the property that any engineer can read the entire policy in one sitting, which matters when an incident is unfolding at 2am.

We do not route on “uncertainty” via a single-model self-reported confidence score. The literature on calibration of these scores is mixed; what we see in practice is that they correlate with verbosity more than with accuracy. We route on intent and rule, score outputs with the eval suite, and add intents to the rules table when we see the suite reveal a class of query that wants a different tier.

We do not silently degrade. Every degradation is logged, alerted, and visible to the operator. The product manager and the compliance lead see the same data we do, in the same dashboards, in real time.

What this gives you

A routing layer built this way buys a few specific properties. Cost stays predictable. Quality stays auditable. Provider concentration risk drops to roughly zero: if any one provider has a six-hour incident, the system serves degraded traffic through the fallback chain and the operator sees the degradation immediately. Upgrades happen on a schedule the eval suite can defend, not on the day a marketing email lands in a slack channel.

It is one file, a config table, and a few hundred lines of glue, and it earns more than anything else we ship except the eval suite. We ship it on day one.