LLM Gateway and Model Routing

Talk to a gateway, not to a vendor.

Talk to us
Runs against
Every model call in your product
Sits between
Your code and every vendor
You get
A routing policy and a failover path

A direct wire is a decision you cannot revisit

A product wired directly to one vendor’s API inherits that vendor’s pricing, availability and roadmap. Changing your mind later is a rewrite.

The cost of that is not the migration. It is the year before it, when a cheaper model would have done the classification step, a stronger one would have fixed the planning step, and neither change was worth opening the code for. Optionality you never use still has to exist before you can use it.

So we put a vendor-neutral gateway in front of every model, and routing, cost and failover stop being architecture and become configuration.

One gateway in front of four classes of model The product makes one kind of call, into a gateway that holds the routing policy, the budgets and caching, and the failover path. From there four lanes leave: a fast low-cost model for classification and extraction, a frontier model for planning and ambiguous reasoning, a coding-optimised model for code generation, and a self-hosted open-weight model for non-regulated workloads. Because the seam holds the policy, swapping a model is a routing change rather than a project. The star marks the gateway: the seam you keep. One interface in front of every model Your product Gateway Swapping a model is a routing change. routing policy, budgets, failover Fast, low-cost: classification Frontier: planning, reasoning Coding-optimised model Self-hosted: non-regulated work One gateway in front of four classes of model The product makes one kind of call, into a gateway that holds the routing policy, the budgets and caching, and the failover path. From there four lanes leave: a fast low-cost model for classification and extraction, a frontier model for planning and ambiguous reasoning, a coding-optimised model for code generation, and a self-hosted open-weight model for non-regulated workloads. Because the seam holds the policy, swapping a model is a routing change rather than a project. The star marks the gateway: the seam you keep. One interface in front of every model Your product Gateway Swapping a model is a routing change. routing policy, budgets, failover Fast, low-cost: classification Frontier: planning, reasoning Coding-optimised model Self-hosted: non-regulated work
The lanes are replaceable; the seam is not. Everything that decides which model runs — policy, budget, fallback — lives in one place you own, rather than in the call sites scattered through your product.

What the gateway gives you

  • Routing by task Fast, low-cost models for classification, extraction and routing; frontier models for planning and ambiguous reasoning; specialists where they win.
  • One evaluation surface Every model judged by the same acceptance, safety and SLA gates, so a switch is a measured decision rather than a bet on a benchmark.
  • Cost and latency control Budgets, caching and fallbacks applied centrally, with the cost of each route visible.
  • Failover A degraded but working path when a provider is rate-limiting or down, rehearsed before you need it.
  • Open-weight options Where licences and data class allow, self-hosted models for non-regulated workloads that can cut inference cost substantially.
  • Usage accounting per route Spend attributed to the task it served, joined to the same identifier the evaluation work uses, so the gateway can answer what each route actually costs.

How the engagement runs

Four stages, each ending in something you keep. No weeks are attached: the length is set by how many call sites the inventory turns up, which is the one thing nobody knows at the start.

  1. Stage 1 Call inventory Every model call the product makes today, by call site, data class and spend. This stage regularly finds calls nobody remembered shipping. You get: a register of every call site, with its data class and what it costs.
  2. Stage 2 Adapter One interface in front of them all, with the differences that break swaps — structured output, tool-call format, context limits — normalised behind it. You get: one interface, and a model change that is a config edit.
  3. Stage 3 Routing policy Task classes mapped to model tiers, with budgets, caching and fallbacks, and the acceptance gates each route has to pass before it carries traffic. You get: a written routing policy, enforced in the gateway.
  4. Stage 4 Cutover and failover drill Traffic moved onto the gateway, and then a provider outage simulated so the degraded path is exercised while someone is watching rather than at three in the morning. You get: traffic on the gateway, and a failover path you have seen work.

What you leave with

  • The call register — what your product actually asks of a model, which is usually more than anyone expects.
  • The adapter — in your codebase, so the next model comparison costs an evaluation run rather than a quarter.
  • The routing policy — task classes, tiers, budgets, fallbacks and the gate each route must pass.
  • A rehearsed failover — and the record of it working.

The method, published

Layer 3 of the production agent stack sets out the routing table this engagement implements, and the reason for the seam: keep model access behind an adapter and vendors can be compared on the same workflow rather than on their marketing.

The gateway is also where the cost per completed task is actually measurable — a route that cannot report what it spent cannot be compared with one that can.

Tell us how many places in your product call a model directly. If the answer is a guess, the inventory is worth doing on its own.

Talk to us

Related services

AI and Agent Engineering

Agent Production Readiness

Assess an agent system against the eight layers a production stack needs, and get a ranked plan to close the gaps.

Tell me more
AI and Agent Engineering

Agent Evaluation and Cost per Task

Instrument what your agents actually complete, and what each completed task costs once retries and review are counted.

Tell me more
AI and Agent Engineering

AI Governance and Data Boundaries

Model registry, routing policy and a hard data boundary, so regulated workloads can use AI and prove how.

Tell me more