Plano — AI-Native Proxy and Data Plane for Agentic Apps

Introduction

Plano is an open-source AI-native proxy and data plane designed for agentic applications. Built in Rust on top of Envoy, it sits between your agents and LLM providers, handling routing, safety enforcement, cost tracking, and observability so developers can focus on agent logic rather than infrastructure plumbing.

What Plano Does

• Routes LLM requests across multiple providers with failover and load balancing
• Enforces safety guardrails and content policies at the proxy layer
• Provides real-time cost tracking and token usage observability
• Supports prompt caching and semantic deduplication to reduce costs
• Offers an OpenAI-compatible API interface for drop-in integration

Architecture Overview

Plano runs as a lightweight Rust binary that extends Envoy's filter chain with AI-specific middleware. Incoming requests pass through a pipeline of filters: authentication, rate limiting, content safety, provider routing, and telemetry. The routing layer uses configurable strategies (cost-optimized, latency-optimized, or capability-based) to select the best backend. Responses flow back through the same filter chain for logging and policy enforcement.

Self-Hosting & Configuration

• Deploy via Docker or as a standalone binary on Linux
• Configure providers, models, and routing rules in a YAML file
• Set API keys for upstream providers via environment variables
• Integrate with Prometheus and OpenTelemetry for metrics export
• Supports horizontal scaling behind a standard load balancer

Key Features

• Multi-provider routing with automatic failover and retries
• Built-in content safety filters configurable per route
• Token-level cost tracking with per-team and per-project attribution
• Semantic caching reduces redundant calls to expensive models
• Sub-millisecond proxy overhead thanks to Rust and Envoy foundation

Comparison with Similar Tools

• LiteLLM — Python proxy, broad provider support; Plano adds Envoy-grade performance and safety filters
• Portkey AI Gateway — managed service focus; Plano is fully self-hosted and open-source
• OpenRouter — routing API; Plano gives you full control of the data plane
• Envoy AI Gateway — lower-level; Plano provides opinionated AI-specific abstractions

FAQ

Q: Does Plano support streaming responses? A: Yes. Server-sent events pass through with per-token observability hooks.

Q: Can I add custom safety filters? A: Yes. Plano supports Wasm-based filter plugins for custom content policies.

Q: What providers are supported? A: OpenAI, Anthropic, Google, Mistral, Azure OpenAI, AWS Bedrock, and any OpenAI-compatible endpoint.

Q: How does semantic caching work? A: Plano embeds request prompts and returns cached responses for semantically similar queries within a configurable similarity threshold.

Sources

• https://github.com/katanemo/plano
• https://github.com/katanemo/plano/blob/main/README.md