LiteLLM — Unified Proxy for 100+ LLM APIs

What it is

LiteLLM is a Python library and proxy server that provides a unified API interface for calling over 100 LLM providers. You write your code once using the OpenAI SDK format, and LiteLLM translates the call to whichever provider you specify: Anthropic Claude, Google Gemini, Azure OpenAI, AWS Bedrock, Ollama, Groq, Together AI, and many more. Switching between providers is a one-line change in the model name.

The library can be used as a Python SDK (direct import), as a standalone proxy server (OpenAI-compatible endpoint), or as a gateway for managing keys, budgets, and rate limits across teams. It is designed for developers and organizations that use multiple LLM providers and want a consistent interface without vendor lock-in.

How it saves time or tokens

Without LiteLLM, calling different providers requires different SDKs, different request formats, and different response parsing. Switching from OpenAI to Claude means rewriting API calls, changing authentication, and adjusting response handling. LiteLLM handles all provider differences behind a single completion() call.

The proxy server mode adds operational value: centralized API key management, per-user budgets, request logging, model fallbacks (try Claude, fall back to GPT-4 if it fails), and load balancing across multiple model deployments. These features save significant engineering time for teams running LLM operations at scale.

How to use

1. Install LiteLLM:

```bash

pip install litellm

```

1. Use as a Python SDK:

```python

from litellm import completion

# Call Claude

response = completion(

model='anthropic/claude-sonnet-4-20250514',

messages=[{'role': 'user', 'content': 'Hello!'}],

)

# Switch to GPT-4 by changing one line

response = completion(

model='gpt-4o',

messages=[{'role': 'user', 'content': 'Hello!'}],

)

```

1. Or run as a proxy server for team-wide access.

Example

Running LiteLLM as a proxy server:

# Start the proxy
litellm --model anthropic/claude-sonnet-4-20250514

# The proxy listens on localhost:4000
# Any OpenAI-compatible client can connect
curl http://localhost:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "anthropic/claude-sonnet-4-20250514",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Proxy configuration with fallbacks and load balancing:

# litellm_config.yaml
model_list:
  - model_name: default
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: sk-ant-...
  - model_name: default
    litellm_params:
      model: gpt-4o
      api_key: sk-...

router_settings:
  routing_strategy: simple-shuffle  # load balance
  num_retries: 2                    # fallback on failure

Related on TokRepo

• AI Gateway Tools — Compare LiteLLM with other LLM gateway and proxy solutions.
• AI Gateway: LiteLLM — Deep dive into LiteLLM configuration on TokRepo.

Common pitfalls

• Not setting the correct environment variable for each provider. LiteLLM reads API keys from environment variables: ANTHROPIC_API_KEY for Claude, OPENAI_API_KEY for OpenAI, GEMINI_API_KEY for Google. Missing keys produce authentication errors that may be confusing when switching providers.
• Assuming all providers support all features. Some providers do not support streaming, function calling, vision inputs, or JSON mode. LiteLLM translates what it can, but if a provider does not support a feature, the call fails. Check the provider's capabilities before relying on advanced features.
• Running the proxy without rate limiting in production. Without rate limits, a single client can exhaust your API budget. Configure per-user or per-team budgets in the proxy config to prevent runaway costs.