Cloudflare AI Gateway — LLM Proxy, Cache & Analytics

What it is

Cloudflare AI Gateway is a free proxy gateway for LLM API calls. It sits between your application and AI providers (OpenAI, Anthropic, Google, etc.), adding response caching, rate limiting, cost tracking, fallback routing, and analytics. You change one URL in your code and get visibility and control over all your AI API usage.

It is designed for engineering teams running production AI applications who need cost control, observability, and resilience without building custom proxy infrastructure.

How it saves time or tokens

The token estimate for this workflow is 2,600 tokens. Response caching can reduce costs by up to 95% for repeated queries. Rate limiting prevents budget overruns. Fallback routing automatically switches to a backup provider if your primary one goes down, improving reliability without code changes.

How to use

1. Sign up at dash.cloudflare.com (free tier available)
2. Navigate to AI > AI Gateway > Create Gateway
3. Replace your API base URL with the Cloudflare gateway URL
4. Your existing code and API keys continue to work

Example

from openai import OpenAI

# Before: direct to OpenAI
# client = OpenAI(base_url='https://api.openai.com/v1')

# After: route through Cloudflare AI Gateway
client = OpenAI(
    base_url='https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_name}/openai'
)
# Same API key, same code -- now with caching, logging, and analytics

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Explain caching'}]
)

# Universal endpoint for any provider
curl https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_name}/openai/chat/completions \
  -H 'Authorization: Bearer sk-...' \
  -H 'Content-Type: application/json' \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'

Related on TokRepo

• Cloudflare AI Gateway Deep Dive -- Detailed configuration guide
• AI Gateway Providers Compared -- Compare all AI gateway options

Common pitfalls

• Caching only works for identical requests; small prompt variations generate cache misses
• The gateway adds a small latency overhead (typically under 50ms) for the proxy hop
• Fallback routing requires configuring multiple providers; it does not auto-discover alternatives