The real cost of model routing has nothing to do with sticker price

Model routing looks clean on a whiteboard. Route simple queries to cheaper, faster models. Keep the heavy models for the hard stuff. But the data center floor is not a whiteboard. In production, routing becomes a constant juggling act, and the variables that dominate cost and latency are almost never the ones engineers list in a comparison table. The same gap between neat theory and messy production reality shows up in AI inference optimization, as DeepSeek's DSpark paper demonstrates for speculative decoding.

This article walks through three dimensions that made routing surprisingly difficult for us, drawing on 417 tasks from the AppWorld Test Challenge with a CodeAct agent. Every figure comes from a real deployment, not a simulation.

Cost Is More Than Model Pricing

The assumption going in was simple: GPT-4.1 should be cheaper than Claude Sonnet 4.6. GPT-4.1 per-token pricing is lower on both input and output, and Sonnet takes roughly three times as many reasoning steps to complete the same agentic tasks. On sticker price alone, GPT-4.1 wins.

The actual numbers told a different story. Across 417 tasks with the same agent, Sonnet cost $79 total ($0.19 per task) while GPT-4.1 cost $155 total ($0.37 per task), nearly double.

The explanation is caching, a factor most routing architectures ignore entirely. Agent workloads reuse large chunks of context across consecutive steps. When cache hit rates are high, effective input costs drop dramatically. Sonnet lower cache-read pricing meant it benefited disproportionately from this pattern, enough to overcome both its higher base pricing and its longer trajectories. A router that looks only at the pricing sheet is optimizing against the wrong numbers. This variable-heavy landscape is why Nvidia's recent work on consolidating model capabilities matters: reducing the number of models in play simplifies the routing problem.

Complexity Is More Than Task Difficulty

A common routing strategy is to measure task difficulty and send harder work to stronger models. Intuitive, but it breaks down in two ways in production.

First, difficulty is often invisible at routing time. A request like "summarize this contract" looks simple, but it might trigger retrieval, compliance checks, tool use, and multiple rounds of refinement before it is done. Meanwhile, a highly technical prompt might be handled efficiently by a smaller specialized model. You often do not know how hard a task is until execution reveals its internal structure.

Second, even if you could estimate difficulty perfectly, it is only one signal among many. Enterprise routers need to balance cost, latency, model specialization, and reliability simultaneously. Regulatory constraints pile on: data residency rules, approved model lists, compliance requirements. A task that would ideally go to one model might need to go elsewhere because of governance, and the router must handle that gracefully rather than fail. This balancing act echoes the tension in multi-agent coordination that group-chat agent research has begun to formalize.

Routers are not solving one problem. They are constantly juggling cost, quality, latency, compliance, and reliability all at once.

Latency Is More Than Model Speed

It is conventional wisdom to think of latency purely in terms of model size: bigger models are slower, smaller ones faster. But what the user experiences depends on much more than that.

Routing itself adds overhead. Infrastructure factors, which hardware a model is running on, whether the cache is warm, how busy the endpoint is, often dominate end-to-end response times. A theoretically faster model can still produce a slower user-facing experience if the serving conditions are not aligned. Recent work on optimizing agent evaluation pipelines by Ai2's olmo-eval shows how infrastructure-level choices can reshape performance metrics.

Then there is routing granularity. Routing once per task adds minimal overhead. But routing at every step, which gives more flexibility to adapt mid-execution, means every additional decision point introduces latency and operational complexity. A router that ignores the serving system is fighting the wrong fight.

An optimization-based approach

These lessons drove a fundamental shift: we stopped treating routing as a classification problem and started treating it as an optimization problem. Instead of asking "which model is best for this task?", the algorithm optimizes simultaneously over cost, quality, and latency, while remaining lightweight enough to avoid becoming a bottleneck itself. This mirrors the approach taken by Nvidia's Nemotron team when they found that synthetic data quality, not model architecture, was the real bottleneck in agentic systems.

On the AppWorld Test Challenge, an optimization-based router traced a clean cost-accuracy frontier. Configuration 1 (latency-optimized) hit 84% accuracy for $93 and 83 seconds. That is a 21% cost reduction and 9% latency reduction compared to running Opus alone, with only a 4% accuracy drop. A standard difficulty-based router landed in a similar accuracy range but at higher cost; it did not explore the full tradeoff space. The optimization itself runs in roughly 6 milliseconds and 2 kilobytes of memory per task.

The key takeaway: routing is not about choosing models. It is about optimizing systems. Models are one variable, an important one, but just one among caching behavior, infrastructure state, compliance constraints, and workload patterns. For teams building the evaluation side of these systems, SkillCoach's process-focused grading offers a complementary approach to understanding when routing decisions are actually working.

If you are building routing into your own agentic systems, the author welcomes hearing about the tradeoffs you are encountering.