Inside theSoftware FactoryWhat is a software factory?
Software factory pattern

The model router

What a model router is, why large teams route every model request through one layer, and what to look for when deciding on one.

What it is

A single internal layer that sits between every agent and every model provider, the same shape as the MCP gateway pattern but applied to model access instead of tool access. An agent asks for a completion; the router decides which provider and model actually serves it, and the calling code never needs to know or care which one that was.

Fig. 1
Agents & tools one call site Model router route · failover · mask Provider A primary if A degrades Provider B failover target Provider C
Every agent and tool calls one router; which provider actually serves the request, and what happens when the primary one degrades, is the router's decision, not the caller's. Hover a box to trace its connections. Original diagram, not a reproduction. Click to enlarge.

Why you need it (in a large team)

  • Betting every agent, tool, and integration directly on one model provider means a single outage, deprecation, or price change reaches all of them at once. No seam anywhere to absorb it. A router turns that into a decision made in one place.
  • Zalando's been running theirs since January 2024, explicitly framed as deliberate vendor independence rather than a convenience.
  • Uber's serves over 800 internal projects through a single point, handling more than 100 million requests a day. Neither reads as an experiment. Both read as infrastructure a large organization concluded it couldn't safely go without.

How it should work

Every request should pass through the same place regardless of which tool or agent sent it. Shopify states this plainly: one internal proxy handles every AI request "regardless of which tool originated it," which is what makes bulk provider leverage and automatic failover possible in the first place. Try to negotiate pricing or handle an outage per-integration instead, and neither is achievable, the same lesson the MCP gateway pattern already teaches about tool access.

The router tends to pick up a second job beyond routing, the same way an MCP gateway tends to pick up policy enforcement beyond translation: Shopify's proxy also does PII masking and prompt-injection detection on every request that passes through it, because it's the one place that sees all the traffic. And because everything is centralized, cost becomes measurable in a way it wasn't before. Uber publishes cost per session and cost per 1,000 requests precisely because the router gives it one place to measure from, not because measurement was the point of building it.

Where it fits into a software factory

Every other pattern that involves an agent making a model call, background agents, automated reviewers, migration scripts, calls through here. It's substrate, not a step in the pipeline: nothing above it changes what it does, but everything above it depends on it working.

What you should look for when deciding on one

  • Whether the router is meant to just route, or also to be where compliance and safety checks live. Shopify bundled PII masking and injection detection onto the same proxy deliberately; a router that only ever forwards requests untouched is a simpler system, but it gives up the one property a shared chokepoint is best positioned to provide.
  • What a provider swap actually costs once the router is in place. None of the sources in this roster describe a real cutover in detail, whether swapping providers behind the router truly requires zero changes to calling code, or just less than it would without one, so treat "swap providers freely" as a design goal to verify, not a benefit to assume.
  • Whether cost visibility is a byproduct of centralizing, or needs separate instrumentation on top. Uber's per-request cost figures exist because the router made them measurable, but measuring wasn't automatic, someone had to build the reporting on top of the chokepoint the router provided.

Companies using a model router

CloudflareAI Gateway routes every request, 241 billion tokens in one 30-day window across multiple providersShopifybulk-provider leverage, automatic failover, PII masking, and injection detection on one proxyStripemulti-provider model access through a shared internal gatewayUbercost per session and cost per 1,000 requests, measurable because everything is centralizedZalandorunning since January 2024, framed explicitly as vendor independence, not convenience

Stripe is tagged multi-provider in our own frontmatter, but nothing in its published entry describes a router, a proxy, or more than one model provider. Likely a mistagging to fix separately, the same issue found on Shopify's mcp-gateway tag while researching that pattern.

← All 9 organisations