Why single-provider was the right answer in 2023 — and isn’t anymore
When the first wave of production LLM applications shipped in 2023, picking a single provider was the right call. The model gap between #1 and #3 on most benchmarks was wide enough that mixing providers added more complexity than it saved. Token prices were high but stable. Latency was bad but uniformly bad. There was no real architectural reason to multi-provider.
In 2026 that calculus has completely inverted. Three things changed:
- The capability gap closed at the top. Claude Opus 4.7, GPT-5.x, and Gemini 2.5 Pro now produce roughly comparable output on most reasoning tasks. Different shapes of mistake, similar quality.
- The price-floor model is now competent. Qwen 3, Llama 4, and DeepSeek V3 can self-host competitively for narrow tasks at 10–30% of frontier API cost.
- Specialization is real. Claude is meaningfully better at code review and long-context refactoring. GPT is meaningfully better at structured output with deeply nested schemas. Qwen is shockingly good at translation and table extraction.
The result is that any application above moderate scale loses money or quality (often both) by routing 100% of traffic to a single provider.
The routing matrix we actually use
Below is the decision matrix we use in production builds. It is not exhaustive and the boundaries move every quarter. But it is honest, and it is what we’d use today for a new build kicked off this week.
Orchestrator agents → Claude Opus
When an agent has to plan, decompose, call tools in sequence, and recover from failure, Claude Opus produces measurably better results than any alternative we’ve tested. The price-per-token is the highest in the market. It is also the only model where we trust the tool-calling reasoning without a second-pass validator. For agent orchestration we use Opus and stop thinking about it.
Worker tasks → Claude Sonnet / GPT-4.x mid-tier
For the actual work the orchestrator dispatches — code edits, document drafts, transformations — Sonnet-class models are the right floor. They are 1/5th the cost of Opus and produce results that are 90–95% as good for narrow, well-specified work. The trick is keeping the work narrow and well-specified. Sonnet is bad at ambiguous decisions; it is excellent at “do this specific thing.”
Structured output with deeply nested schemas → GPT-4.x
If your application requires a strict JSON schema with 8+ levels of nesting and the structure has to be exactly right, GPT’s structured output mode is still the most reliable. Claude is closing the gap and we expect this to flip within the year. For now, GPT.
Long-context analysis → Claude (1M)
For analyzing large codebases, multi-document legal review, and long transcript work, Claude’s 1M context window with prompt caching is the best price/performance available. We do not even benchmark alternatives for this class of work.
High-volume classification → Self-hosted Qwen / Llama
If you’re classifying millions of items a month (support tickets, product descriptions, lead intent), the unit economics push you to a self-hosted small model. Qwen 3 8B fine-tuned on 5–10k labeled examples produces classification accuracy within 2 points of GPT-4 at 5% of the cost. Run it on a single GPU instance and forget about it.
Translation, structured extraction from messy tables → Qwen
For specific narrow tasks — multilingual translation, table extraction from PDF, OCR cleanup — Qwen is currently the best on the market regardless of price. We route to it via API for these tasks and don’t consult anything else.
Fast cheap drafts → Haiku / GPT-mini
For “give me a 50-word summary,” “categorize this thing,” “draft an email subject line” — the cheap tier is now competent enough to use without hesitation. Latency is the win as much as cost.
The architectural pattern
Routing has to live in code, not in the prompt. The pattern we use:
- A thin router layer that takes a task descriptor (task type, expected length, latency budget, cost budget) and returns a provider+model+config.
- A single client interface that hides the provider differences behind a unified call shape. We use MCP for tool definitions to keep them portable across providers.
- Provider-specific shims that handle the wire-format quirks (Anthropic prompt caching vs OpenAI structured output vs OpenRouter for self-hosted).
- A cost meter and a latency budget wrapping every call. Both are first-class observability signals.
- Fallback chains for resilience: if Claude is having an outage, fall back to GPT, then to a self-hosted backup. Production agents cannot have a single point of failure.
The router layer should be 200 lines of code. It is not glamorous. It is the highest-leverage code in the system.
The runtime cost of getting routing wrong
In a recent build we audited, the team had defaulted everything to GPT-5. The monthly bill was $48k. We rerouted:
- Orchestrator calls (5% of traffic, 30% of cost) to Claude Opus → better agent results, slightly more expensive
- Worker drafts (60% of traffic, 50% of cost) to Claude Sonnet → 60% cheaper per-token, equivalent quality
- Bulk classification (30% of traffic, 15% of cost) to self-hosted Qwen → 90% cheaper
- Structured output for one specific endpoint (5% of traffic, 5% of cost) → kept on GPT, it was working
New monthly bill: $19k. Agent quality measurably improved. Total engineering effort to switch: about three days, almost all of it spent on the unified client interface that should have existed from day one.
That’s roughly the size of the prize for any mid-scale application that hasn’t audited its routing in the last six months.
Three things that will be true a year from now
- Routing will be invisible. Open-source routers (LiteLLM, OpenRouter, etc.) will continue to mature and will absorb most of this complexity. You’ll still need to make the routing decisions; the wire-format work will be commodity.
- Provider lock-in clauses will get harder to negotiate. Vendors know that multi-provider routing is the way. Watch your contracts for “minimum spend” clauses that lock you into a single provider’s volume tier.
- Cost meters will be a board-level metric. Token cost as a percentage of revenue is going to show up in finance dashboards next to AWS spend. Programs without good cost meters will get cut first.
The take
Single-provider was a reasonable bet in 2023. Multi-provider is the default in 2026. The architectural cost of doing it right is small. The cost of not doing it is roughly half your inference bill and a meaningful chunk of your output quality.
We do multi-provider routing audits as part of Custom AI Builds and as a standalone engagement. If you’re spending more than $25k/month on inference and haven’t audited the routing in six months, the audit usually pays for itself. Schedule a call.
