MMOOZZGG.com

Glossary

What is an LLM router?

In short

An LLM router (model router) is a program that decides, for each request, which language model should answer it: simple questions go to a cheap and fast model, hard ones to a more powerful model.

Language models · LLM router

MMOOZZGG.com editorsChecked: 1 min read

For a child

At a clinic, you first go to the front desk. They ask what hurts and send you to the right doctor: one for a cold, another for a broken arm. Nobody sends everyone straight to the top professor. Neural networks can have such a front desk too: it decides which of them gets the question.

In plain words

A company may have access to several models: a small cheap one, a large expensive one, and one for images. Sending everything to the large model wastes money, and sending everything to the small one is risky. The router sits at the entrance and quickly rates each request: how hard it is, whether it needs code, whether it has an image, whether the document is long. Based on this, it picks the model. If the answer is weak, some setups pass the question on to the next, stronger model.

For business

Routing cuts costs without a visible loss of quality when most requests are simple. It also lowers dependence on one vendor: if a model fails or its price rises, traffic moves to another model. The costs are building the rules or training a classifier, tests and quality monitoring. The risks: the router makes a mistake and sends a hard question to a weak model, different models answer in different styles, and data may go to a service under another jurisdiction. Before launch, label real requests by difficulty and check where the line between models lies for your task.

For a specialist

An LLM router solves the task of choosing a model for each request, balancing quality, cost and latency. Approaches: rules based on request features; a classifier that predicts whether a weak model will cope; and a cascade, where the cheap model’s answer is checked and the next model is called when confidence is low. FrugalGPT (Chen, Zaharia, Zou, 2023) described three saving strategies: prompt adaptation, model approximation and the cascade; the cascade matched GPT-4 at up to 98% lower cost, or raised accuracy by 4% at the same cost. RouteLLM (Ong et al., 2024) trains the choice between a strong and a weak model on human preference data with augmentation, and in some cases cuts costs by more than half without a loss of quality; trained routers stayed effective when the model pair was replaced. Do not confuse it with the router inside a mixture of experts, which picks experts for each token within one network.

Do not confuse. Do not confuse it with a network router, which passes data packets between computer networks.

Example

An online store sends questions about order status to a small model, and complaints and refund disputes to a large one. The model bill went down, and hard requests get more detailed answers.

Related terms

Questions and answers

What is an LLM router? An LLM router, also called a model router, takes a request, rates it and decides which language model to send it to, so you get the quality you need for less money.
How is a cascade different from a router? A router picks the model before the answer, from the request itself. A cascade first gets an answer from a cheap model, checks it and calls a stronger model when in doubt.

Hard requests take longer in a cascade because they go through two models.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us