# RouteLLM — Smart Router for Cheaper LLM Inference

> Route prompts between strong and weak LLMs to cut costs by up to 85 percent while keeping 95 percent of quality.

## Install

Save as a script file and run:

# RouteLLM — Smart Router for Cheaper LLM Inference

## Quick Use
```bash
pip install "routellm[serve,eval]"

# Use as an OpenAI-compatible client
python -c "
from routellm.controller import Controller
client = Controller(routers=['mf'], strong_model='gpt-4o', weak_model='gpt-4o-mini')
response = client.chat.completions.create(
    model='router-mf-0.11593',
    messages=[{'role': 'user', 'content': 'Hello'}]
)
print(response.choices[0].message.content)
"
```

## Introduction
RouteLLM is a framework from LMSYS that routes LLM queries between expensive and cheaper models based on prompt complexity. Simple queries go to a smaller model while complex ones go to the strongest model, cutting costs significantly without meaningful quality loss. It ships with pre-trained routing models and works as a drop-in replacement for the OpenAI client.

## What RouteLLM Does
- Analyzes each prompt to decide whether a strong or weak model is needed
- Reduces LLM inference costs by up to 85 percent on standard benchmarks
- Provides a drop-in OpenAI-compatible API for seamless integration
- Ships with pre-trained routers so you can start routing without training
- Lets you calibrate the cost-quality tradeoff with a single threshold parameter

## Architecture Overview
RouteLLM works by running a lightweight router model on each incoming prompt. The router outputs a win-rate probability for the strong model. If the probability exceeds a configurable threshold, the request routes to the strong model; otherwise it goes to the weak model. The framework includes several router implementations: matrix factorization (MF), BERT-based classifiers, LLM-based judges, and weighted ranking models. All routers are trained on preference data from the Chatbot Arena.

## Self-Hosting & Configuration
- Install via pip with optional serve and eval dependencies
- Configure strong and weak model endpoints (any OpenAI-compatible API)
- Set the cost threshold in the model name string (e.g., `router-mf-0.11593`)
- Launch as an OpenAI-compatible server with `python -m routellm.openai_server`
- Calibrate thresholds using your own evaluation data for domain-specific routing

## Key Features
- Multiple pre-trained router types including MF, BERT, and LLM-based options
- Threshold calibration tool that maps cost-quality curves to your use case
- OpenAI-compatible API server mode for production deployments
- Evaluation framework to benchmark routing quality on standard datasets
- Extensible architecture for adding custom router implementations

## Comparison with Similar Tools
- **Manual model selection** — requires developers to choose per-request; not scalable
- **Random load balancing** — distributes by load, not complexity; wastes spend on easy queries
- **OpenRouter** — commercial routing service; RouteLLM is self-hosted and open source
- **Portkey AI Gateway** — proxy with fallback routing; RouteLLM uses quality-aware routing
- **Martian** — commercial smart routing; RouteLLM provides the research and code openly

## FAQ
**Q: What models can I route between?**
A: Any two models accessible via an OpenAI-compatible API. Common pairs include GPT-4o / GPT-4o-mini and Claude Sonnet / Claude Haiku.

**Q: Do I need to train my own router?**
A: No. RouteLLM ships with pre-trained routers. You can optionally fine-tune them on your own preference data.

**Q: How do I choose the right threshold?**
A: Use the calibration tool to plot cost vs. quality on a sample of your traffic. Pick the threshold that matches your budget.

**Q: Does RouteLLM add latency?**
A: The router inference adds a few milliseconds. For most applications this is negligible compared to LLM generation time.

## Sources
- https://github.com/lm-sys/RouteLLM
- https://lmsys.org/blog/2024-07-01-routellm/

---
Source: https://tokrepo.com/en/workflows/asset-59ec662e
Author: Script Depot