MMOOZZGG.com

LLM

What is an LLM router, and how does it cut your AI bill?

8 min read6 sources

An LLM router (model router) is a program that picks a model for each request: it sends a simple question to a fast, low-cost model and a hard one to a strong model. In the 2024 RouteLLM study, this choice cut costs by more than 85% on the MT Bench test while keeping 95% of GPT-4 quality. A router usually sits next to a gateway (LLM gateway): one address for all models, keys, budgets and a backup model if one fails.

On this page · 7 sections

Router, gateway and proxy: what is the difference

The three words are often mixed up, though they mean different things. A router (model router) decides which model gets a specific request. A gateway receives all of a company’s requests at one point and only then passes them to the models.

A proxy in this topic is almost a synonym of a gateway: a program that sits between your application and the model providers. For example, LiteLLM calls one and the same product both AI Gateway and LiteLLM Proxy.

A router usually works inside a gateway or next to it: the gateway gives one entry, keys and cost tracking, and the router adds the choice of model. It has nothing to do with a network router that passes data packets between networks.

An API aggregator works differently: it is someone else’s service that sells access to many models through its own account. How it differs and when it is convenient is covered on the page about API aggregators.

How a router decides which model gets a request

The path of a request is shown in the diagram below. The request arrives at the gateway: one address, keys, budgets. The gateway passes it to the router, and the router answers one question: is the request simple or hard?

If it is simple, the request goes to a fast, low-cost model: it can handle a reference question, an order status or a short translation on its own. If it is hard, a strong model gets it: a contract review, a calculation, a task in several steps.

If the model provider does not answer, the gateway moves the request to a backup model. In the diagram this is the dotted line “Failure: backup model”.

A router judges difficulty in two ways. The simple way is rules: the length of the request, keywords, whether code or a file is needed. The more accurate way is a small classifier model that predicts whether the weak model will cope.

In RouteLLM, routers were trained on human preference data: which of two answers a person chose. There is also a cascade: a low-cost model answers first, the answer is checked, and in case of doubt the strong model is called, but then a hard request passes through two models and takes longer.

How much it saves: research numbers

The main numbers come from RouteLLM, an open framework by LMSYS researchers published on 1 July 2024. The router chose between the strong model GPT-4 Turbo and the cheaper Mixtral 8x7B.

At 95% of GPT-4 quality, costs fell by more than 85% on MT Bench, by 45% on MMLU and by 35% on GSM8K. MT Bench is a set of multi-turn conversation questions, MMLU tests knowledge across many subjects, and GSM8K is school math problems.

The range from 35% to 85% shows the main point: the saving depends on the share of simple requests. Where there are many hard tasks, the router calls the strong model more often, and the gain is smaller.

In the paper on the framework, the authors are more careful: in a number of cases costs fall more than 2 times with no loss of quality. The routers kept the effect even when the pair of models was replaced without retraining.

You can work out your own case from the price of tokens, the pieces of words you pay the provider for. How to count tokens is covered on the page about token costs, and the cloud API calculator helps you estimate the monthly bill.

A router inside the models themselves

ChatGPT has a router inside too. In the GPT-5 system card (August 2025), OpenAI describes a system of a fast model, a reasoning model for hard tasks and a real-time router.

The router decides by the type of conversation, its difficulty, the tools needed and an explicit request from the user, such as “think hard about this”. It learns from real signals: when people switch models themselves and which answers they rate higher.

For a company this means two things. First, large providers already use a router in their products. Second, inside someone else’s service you do not control the choice of model and the price of each answer, while your own router gives you that control.

Do not confuse this with the router inside a mixture of experts: there the choice happens inside one neural network for each token, not between separate models.

What a gateway gives: one address, budgets, a backup model

You need a gateway as soon as several departments use models. Take LiteLLM: it is an open gateway that you install on your own server. According to its documentation, it gives one address in the OpenAI format for 100+ model providers.

The OpenAI format means that a program written for the OpenAI API works with any model behind the gateway without rewriting code. More on the page about the OpenAI-compatible API.

Each user, team or project gets its own virtual key with a list of models, a budget and limits. The gateway records the cost of each request, and costs are visible by key, team and user.

When a provider fails, the gateway moves requests to another model or retries. Request logs go to monitoring systems, and filters can hide personal data before it is sent to a model.

The LiteLLM code is open under the MIT license, except for the enterprise folder: the features in it come under a separate commercial license. If you need those features, check the terms before rollout.

A setup for a company: cloud, your own model, 152-FZ

A router solves not only the question of price but also the question of data. Under 152-FZ, Russia’s personal data law (Part 5 of Article 18), personal data of Russian citizens may not be recorded and stored in databases outside the country, except in cases the law names directly.

This gives a working setup. The cloud model gets only requests without personal data: general questions, website texts, code. Everything with customers’ names, phone numbers and contracts goes to your own model inside your network.

In this setup the router also checks each request for personal data, and the gateway on your server keeps a log of which request went where.

How the hybrid setup works and what your own part costs is covered on the page about the hybrid setup. The legal side is described in detail on the page about the law and data.

When you do not need a router

If you have one application, one model and one developer, a gateway is extra. The LiteLLM documentation itself advises a plain Python library in that case, and a gateway when several people or programs send requests and you need keys, cost tracking or filters.

You do not need a router if almost all requests are of one type. When all tasks are hard, the strong model gets them anyway, and when all are simple, one low-cost model is enough.

One more case: there are few requests. The saving is a percentage of the bill, and with a small bill it will not pay for the setup and quality checks.

Start with a measurement: export a month of requests, label them by difficulty and count what share could go to a low-cost model.

Questions and answers

Does a router make answers worse?

It can, if it makes a mistake and gives a hard request to a weak model. In RouteLLM the threshold was set to keep 95% of GPT-4 quality, and even so costs fell several times. Test the router on your own requests before launch and watch the share of complaints about answers.

How is an LLM gateway different from an API aggregator?

A gateway is your own program: you install it on your server, connect your own provider keys and see the costs and logs yourself. An aggregator is someone else’s service that sells access to many models through its own account. Requests through an aggregator pass through its servers, and that matters for personal data.

Can you run a router on your own servers?

Yes. LiteLLM installs on your own server, and the RouteLLM code is open. Behind the gateway there can be both cloud models and your own model inside the company network.

Sources and fact-checking

Checked by the editors on 16 October 2026.

  1. LMSYS, 1 July 2024 (lmsys.org)
  2. OpenAI, arXiv, 19 December 2025 (arxiv.org)
  3. LiteLLM Docs, (docs.litellm.ai)
  4. GitHub, BerriAI/litellm, LICENSE, 2023 (github.com)
  5. arXiv, RouteLLM, 26 June 2024 (arxiv.org)
  6. ConsultantPlus, 152-FZ, Article 18, 2026 (consultant.ru)

Found a mistake? Tell us

Publications

Mike Mozg

Agentic AI use cases at AT&T: nearly 500 processes now run on agents

AT&T no longer uses AI only to help staff work faster: it is rebuilding whole business processes around agents. SiliconANGLE reported on 14 November 2026 that nearly 500 agentic AI use cases at AT&T are already in production. Here is how the company rebuilds them, where a human still checks the agent, and what other large companies can take from it.

News4 min
Contact us