LLM Observability

What's in this pack

You can't fix what you can't see. The day a prompt regression silently 3x's your token bill is the day you wish you'd installed an observability layer last quarter. This pack collects the seven assets that turn an opaque LLM black box into a debuggable, alertable, optimizable system.

Why this matters

Three production failure modes that observability catches and intuition misses:

1. Silent token inflation. A "minor" prompt edit adds a 200-token reminder. Multiply by 1M requests/day and that's $2-6k/mo extra you didn't budget for. Langfuse's per-prompt-version cost view surfaces it on day one.
2. The 95th-percentile tail. Average latency looks fine — but the 5% of queries hitting cold cache, retry loops, or oversized RAG payloads tank user experience. p99 dashboards from Phoenix or LangSmith make the tail visible.
3. Quality regression invisible at the unit level. Each individual response looks plausible. Aggregate evaluator scores (LLM-as-judge, retrieval recall, hallucination rate) over the last 24h vs the previous 7d, and the regression jumps out.

Install in one command

# Install the entire pack
tokrepo install pack/llm-observability

# Or pick the platform you want to start with
tokrepo install langfuse
tokrepo install agentops
tokrepo install phoenix

The TokRepo CLI drops the SDK config and dashboard scaffolding into your project so traces start flowing on the next request — no manual instrumentation walk-through required.

Common pitfalls

• Logging full prompts and PII to a third-party SaaS. If your prompts include user data, self-host Langfuse or Phoenix; don't ship raw payloads to LangSmith Cloud without redaction. All three open-source options run on a single VM under 4GB RAM for typical loads.
• No sampling on high-volume endpoints. Tracing 100% of requests at 1M/day will overwhelm both your storage and your wallet. Sample 10% by default, 100% on errors. Langfuse and Phoenix both support this natively.
• Tracking tokens but not dollars. Different models price differently per token. Configure model-pricing in your platform once; track cost in dollars, not just token counts. CFOs care about dollars.
• One generic dashboard for everyone. Build one dashboard per persona — eng (latency, error rate), product (cost per feature), exec (cost per active user, week-over-week trend). Generic dashboards get ignored.
• No alert on prompt-version cost delta. Add an alert that fires when a new prompt version's avg-cost-per-call deviates >20% from the previous version. This is the single highest-ROI alert you'll set up.

Relationship to other packs

LLM Observability is the runtime telemetry layer. The complementary LLM Eval & Guardrails pack is the offline scoring layer — DeepEval, Promptfoo, Ragas. You want both: observability shows you what's happening in production, eval tells you whether a proposed change is better before you ship.

Multi-Agent Frameworks (CAMEL, LangGraph, DeepAgents) are the systems being instrumented. If you're running a LangGraph workflow and can't see which node failed, you don't have observability — you have a print-statement debugger. Pair the framework pack with this one from day one.