RouteLLM — Smart Router for Cheaper LLM Inference

Introduction

RouteLLM is a framework from LMSYS that routes LLM queries between expensive and cheaper models based on prompt complexity. Simple queries go to a smaller model while complex ones go to the strongest model, cutting costs significantly without meaningful quality loss. It ships with pre-trained routing models and works as a drop-in replacement for the OpenAI client.

What RouteLLM Does

• Analyzes each prompt to decide whether a strong or weak model is needed
• Reduces LLM inference costs by up to 85 percent on standard benchmarks
• Provides a drop-in OpenAI-compatible API for seamless integration
• Ships with pre-trained routers so you can start routing without training
• Lets you calibrate the cost-quality tradeoff with a single threshold parameter

Architecture Overview

RouteLLM works by running a lightweight router model on each incoming prompt. The router outputs a win-rate probability for the strong model. If the probability exceeds a configurable threshold, the request routes to the strong model; otherwise it goes to the weak model. The framework includes several router implementations: matrix factorization (MF), BERT-based classifiers, LLM-based judges, and weighted ranking models. All routers are trained on preference data from the Chatbot Arena.

Self-Hosting & Configuration

• Install via pip with optional serve and eval dependencies
• Configure strong and weak model endpoints (any OpenAI-compatible API)
• Set the cost threshold in the model name string (e.g., router-mf-0.11593)
• Launch as an OpenAI-compatible server with python -m routellm.openai_server
• Calibrate thresholds using your own evaluation data for domain-specific routing

Key Features

• Multiple pre-trained router types including MF, BERT, and LLM-based options
• Threshold calibration tool that maps cost-quality curves to your use case
• OpenAI-compatible API server mode for production deployments
• Evaluation framework to benchmark routing quality on standard datasets
• Extensible architecture for adding custom router implementations

Comparison with Similar Tools

• Manual model selection — requires developers to choose per-request; not scalable
• Random load balancing — distributes by load, not complexity; wastes spend on easy queries
• OpenRouter — commercial routing service; RouteLLM is self-hosted and open source
• Portkey AI Gateway — proxy with fallback routing; RouteLLM uses quality-aware routing
• Martian — commercial smart routing; RouteLLM provides the research and code openly

FAQ

Q: What models can I route between? A: Any two models accessible via an OpenAI-compatible API. Common pairs include GPT-4o / GPT-4o-mini and Claude Sonnet / Claude Haiku.

Q: Do I need to train my own router? A: No. RouteLLM ships with pre-trained routers. You can optionally fine-tune them on your own preference data.

Q: How do I choose the right threshold? A: Use the calibration tool to plot cost vs. quality on a sample of your traffic. Pick the threshold that matches your budget.

Q: Does RouteLLM add latency? A: The router inference adds a few milliseconds. For most applications this is negligible compared to LLM generation time.

Sources

• https://github.com/lm-sys/RouteLLM
• https://lmsys.org/blog/2024-07-01-routellm/