Cutting AI cost with model routing
Most production AI traffic is easy. Sending all of it to your largest model is the most common and most expensive mistake we see.
When a feature graduates from prototype to production, the invoice arrives about a month later. The usual reaction is to negotiate pricing. The usual fix is to stop sending trivial work to an expensive model.
If you look at real traffic, the distribution is lopsided. A large share of requests are classification, routing, extraction or short rewrites — tasks a small fast model handles at parity. The genuinely hard reasoning is a minority of calls but most of the perceived quality.
Routing means classifying the request first and choosing the model deliberately. The classifier itself can be a small model, or often just rules over metadata you already have. What matters is that the decision is explicit and measurable rather than a default.
You cannot do this safely without an evaluation suite, which is the real reason most teams do not. If you cannot score the small model against the large one on your own traffic, switching is a gamble. With a golden dataset it is a measurement that takes an afternoon.
Prompt caching and batching are the other half. Long system prompts re-sent on every call are pure waste, and read-heavy agent workloads batch well. Together with routing, we typically see cost per task fall by around forty percent with no measurable quality change.
The rule we apply: optimise cost only after quality is measured, never before. Cheap and wrong is not a saving, it is a support ticket with a delay.