AgentOps — Observability Dashboard for AI Agents

What it is

AgentOps is a Python SDK for monitoring AI agent sessions with real-time dashboards. It tracks token usage, costs, latency, tool calls, and errors across agent runs. Integration requires adding just two lines of code to your existing agent. The dashboard provides session replay, allowing you to step through an agent's decisions and tool invocations to debug failures.

AgentOps targets developers building AI agents who need visibility into what their agents are doing, how much they cost, and where they fail. It works with any Python agent framework including LangChain, CrewAI, AutoGen, and custom implementations.

How it saves time or tokens

Without observability, debugging a failing agent means adding print statements and re-running sessions. AgentOps records every LLM call, tool invocation, and state transition automatically. The session replay lets you pinpoint exactly where an agent went wrong without re-executing. Token and cost tracking across sessions helps identify which agent patterns are expensive and which are efficient.

How to use

1. Install the SDK:

pip install agentops

1. Add two lines to your agent code:

import agentops
agentops.init('YOUR_API_KEY')  # Get free key at agentops.ai

# Your existing agent code runs unchanged
# AgentOps auto-instruments LLM calls and tool invocations

1. View sessions in the AgentOps dashboard at app.agentops.ai.

Example

Instrumenting a LangChain agent:

import agentops
from langchain.agents import create_openai_tools_agent

agentops.init('YOUR_API_KEY')

# Build your agent as usual
agent = create_openai_tools_agent(llm, tools, prompt)
result = agent.invoke({'input': 'Analyze the latest sales data'})

# AgentOps automatically captures:
# - Each LLM call with tokens and latency
# - Tool calls with inputs/outputs
# - Total session cost
# - Any errors or retries

agentops.end_session('Success')

Related on TokRepo

• AI Gateway Providers — LLM observability and tracing platforms
• AI Tools for Agents — frameworks and tools for building AI agents

Common pitfalls

• AgentOps sends telemetry data to its cloud dashboard; for sensitive workloads, check the data retention policy and consider self-hosted alternatives
• Auto-instrumentation may not capture custom tool calls; use the @agentops.record decorator to manually instrument non-standard tools
• Free tier has session limits; high-volume agent deployments should plan for the paid tier or batch sessions