Portkey AI Gateway — Route to 250+ LLMs

What it is

Portkey AI Gateway is an open-source API gateway that sits between your application and LLM providers. It routes requests to over 250 LLMs with sub-1ms latency overhead. Built-in features include 40+ guardrails, automatic retries, provider fallbacks, semantic caching, and request logging. It has over 11.1K GitHub stars and is Apache 2.0 licensed.

Portkey targets teams running production LLM applications who need reliability, cost control, and provider flexibility without vendor lock-in.

How it saves time or tokens

Portkey's semantic caching returns cached responses for similar queries, saving both tokens and latency. Automatic fallbacks switch to backup providers when the primary is down, avoiding downtime. The unified API means you switch providers by changing a config, not your code.

How to use

1. Run locally:

npx @portkey-ai/gateway

1. Or use the Python SDK:

pip install portkey-ai

1. Route requests through the gateway:

from portkey_ai import Portkey

client = Portkey(api_key='your-key')
response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Hello'}]
)

Example

# Run the gateway locally
npx @portkey-ai/gateway
# API at http://localhost:8787/v1

# Use with curl
curl http://localhost:8787/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'x-portkey-provider: openai' \
  -H 'Authorization: Bearer sk-...' \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'

Related on TokRepo

• AI Gateway Providers — Portkey deep-dive and comparisons
• AI Tools for API — API management and LLM routing tools

Key considerations

When evaluating Portkey AI Gateway for your workflow, consider the following factors. First, assess whether your team has the technical prerequisites to adopt this tool effectively. Second, evaluate the maintenance burden against the productivity gains. Third, check community activity and documentation quality to ensure long-term viability. Integration with your existing toolchain matters more than feature count alone. Start with a small pilot project before rolling out across the organization. Monitor resource usage during the initial adoption phase to identify bottlenecks early. Document your configuration decisions so team members can onboard independently.

Common pitfalls

• Self-hosted gateway requires proper network configuration to reach all LLM provider endpoints.
• Caching aggressive prompts may return stale responses; configure cache TTL based on your use case.
• Guardrails add latency; profile your specific guardrail combination to ensure acceptable response times.