# LiteLLM Proxy — Unified Gateway for 100+ LLM APIs

> LiteLLM Proxy maps 100+ LLM providers (Anthropic, OpenAI, Bedrock, Vertex) to one OpenAI-compatible endpoint. Auth, rate limit, cost track, fallbacks.

## Install

Copy the content below into your project:

## Quick Use

1. Save the YAML config below as `config.yaml`
2. `docker run -p 4000:4000 -v $(pwd)/config.yaml:/app/config.yaml ghcr.io/berriai/litellm:main-stable --config /app/config.yaml`
3. Point any OpenAI SDK at `http://localhost:4000`

---

## Intro

LiteLLM Proxy is a self-hostable gateway that exposes 100+ LLM providers as one OpenAI-compatible endpoint. Point any OpenAI SDK at the proxy and route to Anthropic / Bedrock / Vertex / Together / Groq / local Ollama with no code changes. Adds team-level auth, rate limits, cost tracking, and fallbacks. Best for: enterprises with multi-team / multi-provider LLM use, or anyone who wants to swap models without rewriting clients. Works with: any OpenAI SDK (Python, Node, Go, etc), Claude Code, Cursor (via custom OpenAI base URL). Setup time: 5 minutes (Docker compose).

---

### Run the proxy

```yaml
# config.yaml
model_list:
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: codestral
    litellm_params:
      model: ollama/codestral
      api_base: http://localhost:11434

general_settings:
  master_key: sk-1234
  database_url: postgresql://...

router_settings:
  routing_strategy: simple-shuffle
  fallbacks:
    - claude-3-5-sonnet: ["gpt-4o"]  # if Claude rate-limits, try GPT
```

```bash
docker run -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -e ANTHROPIC_API_KEY \
  -e OPENAI_API_KEY \
  ghcr.io/berriai/litellm:main-stable \
  --config /app/config.yaml
```

### Use from any OpenAI SDK

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-virtual-key-for-team-acme",  # generated via /key/generate
)

# Same API, but routed through LiteLLM
resp = client.chat.completions.create(
    model="claude-3-5-sonnet",   # name from config.yaml
    messages=[{"role": "user", "content": "Hello"}],
)
```

### Per-team budgets and rate limits

```bash
# Generate a key for team Acme with $50/mo budget
curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-1234" \
  -d '{"team_id": "acme", "max_budget": 50, "rpm_limit": 100}'

# Returns: {"key": "sk-acme-xyz123", ...}
```

LiteLLM tracks every call per team, blocks once the budget is hit, and exposes a /spend dashboard.

### Connect Claude Code / Cursor

```bash
# Claude Code
export ANTHROPIC_BASE_URL=http://localhost:4000
export ANTHROPIC_API_KEY=sk-acme-xyz123

# Cursor — Settings > Custom OpenAI Base URL: http://localhost:4000/v1
```

---

### FAQ

**Q: Is LiteLLM Proxy free?**
A: Yes — open-source under MIT license. Self-host with Docker for free. BerriAI also offers a hosted version (LiteLLM Cloud) with SLAs and managed observability for teams that don't want to run infra.

**Q: How does this differ from OpenRouter or Portkey?**
A: OpenRouter is a hosted-only proxy. Portkey is a hosted gateway with observability. LiteLLM is the only one that's primarily *open-source self-host* — full control over auth, routing, and data. All three speak OpenAI format.

**Q: Will my non-OpenAI providers really 'just work' through OpenAI SDK?**
A: Yes for chat/completions. Tool calls work too (LiteLLM normalizes the format). Edge cases: streaming usage stats and provider-specific extensions (Anthropic prompt caching, OpenAI logprobs) need explicit support — most are wired up. Check litellm.ai/docs/providers for your provider.

---

## Source & Thanks

> Built by [BerriAI](https://github.com/BerriAI). Licensed under MIT.
>
> [BerriAI/litellm](https://github.com/BerriAI/litellm) — ⭐ 17,000+

---

<!-- ZH -->

## 快速使用

1. 把下面的 YAML 配置保存为 `config.yaml`
2. `docker run -p 4000:4000 -v $(pwd)/config.yaml:/app/config.yaml ghcr.io/berriai/litellm:main-stable --config /app/config.yaml`
3. 把任何 OpenAI SDK 指向 `http://localhost:4000`

---

## 简介

LiteLLM Proxy 是一个可自托管的网关，把 100+ LLM provider 露出成一个 OpenAI 兼容端点。任何 OpenAI SDK 指向 proxy，就能不改一行代码路由到 Anthropic / Bedrock / Vertex / Together / Groq / 本地 Ollama。带团队级鉴权、限速、成本追踪、自动回退。适合多团队 / 多 provider 的 LLM 用法，或者任何想换模型不改客户端的场景。兼容任何 OpenAI SDK（Python / Node / Go 等）、Claude Code、Cursor（通过自定义 OpenAI base URL）。装机时间 5 分钟（Docker compose）。

---

### 启动 proxy

```yaml
# config.yaml
model_list:
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: codestral
    litellm_params:
      model: ollama/codestral
      api_base: http://localhost:11434

general_settings:
  master_key: sk-1234
  database_url: postgresql://...

router_settings:
  routing_strategy: simple-shuffle
  fallbacks:
    - claude-3-5-sonnet: ["gpt-4o"]  # Claude 限速时回退到 GPT
```

```bash
docker run -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -e ANTHROPIC_API_KEY \
  -e OPENAI_API_KEY \
  ghcr.io/berriai/litellm:main-stable \
  --config /app/config.yaml
```

### 用任何 OpenAI SDK 调

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-virtual-key-for-team-acme",  # 通过 /key/generate 生成
)

# 同一套 API，被 LiteLLM 路由
resp = client.chat.completions.create(
    model="claude-3-5-sonnet",   # config.yaml 里的名字
    messages=[{"role": "user", "content": "Hello"}],
)
```

### 团队级预算和限速

```bash
# 给 Acme 团队生成一个 key，月预算 $50
curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-1234" \
  -d '{"team_id": "acme", "max_budget": 50, "rpm_limit": 100}'

# 返回：{"key": "sk-acme-xyz123", ...}
```

LiteLLM 按团队跟踪每次调用、超预算自动拦、提供 /spend 仪表盘。

### 接 Claude Code / Cursor

```bash
# Claude Code
export ANTHROPIC_BASE_URL=http://localhost:4000
export ANTHROPIC_API_KEY=sk-acme-xyz123

# Cursor —— Settings > Custom OpenAI Base URL: http://localhost:4000/v1
```

---

### FAQ

**Q: LiteLLM Proxy 免费吗？**
A: 免费 —— MIT 开源。自己用 Docker 跑免费。BerriAI 也提供托管版（LiteLLM Cloud）带 SLA 和托管可观测性，给不想自己跑基础设施的团队。

**Q: 跟 OpenRouter 或 Portkey 啥区别？**
A: OpenRouter 只有托管。Portkey 是带可观测性的托管网关。LiteLLM 是唯一主打 *开源自托管* 的 —— 鉴权、路由、数据完全自己控制。三个都说 OpenAI 格式。

**Q: 非 OpenAI provider 真能通过 OpenAI SDK 用吗？**
A: 聊天/completions 行。工具调用也行（LiteLLM 帮你规整格式）。边缘情况：流式 usage stats 和 provider 专属扩展（Anthropic prompt caching、OpenAI logprobs）需要显式支持，多数都接好了。看 litellm.ai/docs/providers 查你的 provider。

---

## 来源与感谢

> Built by [BerriAI](https://github.com/BerriAI). Licensed under MIT.
>
> [BerriAI/litellm](https://github.com/BerriAI/litellm) — ⭐ 17,000+


---
Source: https://tokrepo.com/en/workflows/litellm-proxy-unified-gateway-for-100-llm-apis
Author: LiteLLM (BerriAI)