Phoenix — Open Source AI Observability

What it is

Phoenix is an open-source AI observability platform by Arize AI. It traces LLM application calls, evaluates output quality, and helps debug issues in retrieval-augmented generation (RAG) pipelines, agents, and chat applications. It uses OpenTelemetry for instrumentation and provides a web UI for exploring traces.

Phoenix targets ML engineers and developers building production LLM applications who need visibility into what their AI is doing, why it fails, and how to improve it.

How it saves time or tokens

Phoenix shows you exactly which prompts, retrievals, and tool calls happened in each request. When an LLM produces a bad answer, you trace the root cause (wrong documents retrieved, poor prompt, hallucination) without adding debug logging manually.

The evaluation framework lets you score outputs automatically, catching quality regressions before users report them.

How to use

1. Install Phoenix: pip install arize-phoenix
2. Start the Phoenix server: phoenix serve
3. Instrument your LLM application with the Phoenix OpenTelemetry integration
4. Open the web UI at http://localhost:6006 to explore traces

Example

import phoenix as px
from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor

# Start Phoenix
px.launch_app()

# Register OpenTelemetry tracer
tracer_provider = register(project_name='my-app')

# Instrument OpenAI calls
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

# Your LLM calls are now traced automatically
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'What is RAG?'}]
)

Related on TokRepo

• Phoenix on AI gateway -- Phoenix as an observability provider
• Monitoring tools -- Application monitoring and observability

Common pitfalls

• Phoenix traces can grow large for high-throughput apps; configure sampling rates for production
• OpenTelemetry instrumentation adds slight latency; benchmark before deploying to latency-sensitive endpoints
• The evaluation framework requires labeled data or LLM-as-judge setup; plan your eval strategy before instrumenting