Observabilidad + Tracing de Agentes

What's in this pack

The day an agent silently loops between two tools for 47 turns and returns a confident wrong answer is the day you wish you had span-level traces, not a per-prompt cost chart. This pack is built for the ML/LLM engineer trying to reconstruct what an agent actually did: which sub-agent fired, what arguments went into each tool call, how many retries it ate, what the planner thought before it pivoted.

How this is different from the LLM Observability pack

If you're not sure which pack to install: LLM Observability is the runtime telemetry layer — token cost, p95 latency, error rate, prompt-version dashboards. The audience is anyone shipping LLM calls to production. Agent Observability + Tracing is the debugging layer for systems where one user request fans out into 10–100 LLM calls, tool invocations, and sub-agent handoffs. The audience is the engineer staring at a 4-minute agent run that returned garbage and trying to figure out which step lied.

A cost dashboard tells you the bill went up 30%. A deep trace tells you the planner sub-agent retried the same search 8 times because the tool returned an empty array and the prompt didn't handle it. You want both, but they answer different questions.

Install in a deliberate order

# Full pack
tokrepo install pack/agent-observability-tracing

# Or layer it up
tokrepo install langfuse        # 1. trace store + UI
tokrepo install opentelemetry-llm  # 2. instrument once, swap backends later
tokrepo install phoenix         # 3. eval bridge for trace-level scoring

Five layers, install in this order:

1. Instrumenter — wrap your LLM SDK and agent framework. OpenTelemetry + OpenInference semantic conventions is the right default: instrument once, swap backends without code changes. If you're LangChain-native, the built-in langchain_core.tracers writes directly to LangSmith / Langfuse.
2. Trace store + UI — pick Langfuse self-hosted (data sovereignty), LangSmith Cloud (zero ops, LangChain-tight), Phoenix local (notebook-first, no infra), or Helicone proxy (no SDK install at all). They all consume OTel spans now.
3. Eval bridge — wire your offline evals (LLM-as-judge, retrieval recall, tool-call correctness) into the same trace store so quality scores land on every span. Phoenix and Langfuse both ship this; LangSmith calls it 'feedback'.
4. Alerts — fire on trace-level anomalies, not just per-call ones: agent ran >20 steps, retry depth >5, sub-agent never called expected tool, planner output didn't include required schema. These are the failure modes a per-prompt dashboard misses entirely.
5. Session replay — AgentOps and Helicone both group spans into 'sessions' (one user request = one session). For multi-agent systems this is non-negotiable. Without it you cannot tell two simultaneous user runs apart in the timeline.

Common pitfalls

• Tracing the LLM call but not the tool call. The model emits a tool call; your code runs the tool; the result feeds the next LLM turn. If you only instrument the LLM SDK, the tool execution is a black hole. Wrap your tool dispatcher with the same tracer.
• No parent_span_id on sub-agent handoffs. If sub-agent B is spawned by agent A, B's spans must carry A's trace ID. Otherwise the UI shows two disconnected timelines and you cannot answer 'who called whom'.
• Logging full reasoning chains as a single blob. A reflection loop with 30 thoughts shouldn't be one giant string field — it should be 30 sibling spans under a reflection parent. Filtering, search, and diff all break on the blob shape.
• Sampling agent traces uniformly. Sample 10% of normal runs, but always keep 100% of runs where the trace exited with an error, hit max retries, or had eval score below threshold. The bugs you need to debug are exactly the runs you'd otherwise drop.
• Vendor-locking your spans. Use OpenInference / OpenTelemetry GenAI semantic conventions. Every backend in this pack speaks them. Hand-rolling proprietary JSON means rewriting your instrumentation when you migrate.
• No prompt-version → trace-version link. When a trace was generated by prompt version 7 but you've since shipped version 9, the trace UI must surface the version so you can diff old vs new behavior. Langfuse and LangSmith both support this; wire it on day one.

Pair with these packs

Agent Observability is the debugger. The LLM Observability pack is the production dashboard. The Multi-Agent Frameworks pack is the system being traced (LangGraph, CrewAI, AutoGen). The LLM Eval & Guardrails pack is the scoring engine that turns raw traces into quality signals on the same dashboard. Real teams run all four together — observability without eval is just pretty timelines, eval without traces is just averages.