LangSmith — Prompt Debugging and LLM Observability

What it is

LangSmith is LangChain's observability and evaluation platform for LLM applications. It provides trace visualization for every LLM call, a prompt playground for rapid iteration, dataset-driven evaluation, and regression testing. You see exactly what prompts were sent, what the model returned, how long it took, and how many tokens it consumed.

This tool is for developers building LLM-powered applications who need visibility into model behavior. It works with LangChain, LangGraph, and standalone LLM calls.

How it saves time or tokens

Without observability, debugging LLM applications is guesswork. LangSmith shows the full trace of every chain, agent, or tool call, making it easy to spot where things go wrong. The prompt playground lets you test variations without redeploying. Dataset evaluation automates regression testing across model or prompt changes. The estimated token cost for the monitoring workflow is around 4,100 tokens.

How to use

1. Create a LangSmith account and get an API key.
2. Set the environment variables in your application.
3. Traces are automatically captured for LangChain applications.
4. View traces, run evaluations, and iterate in the dashboard.

# Set environment variables
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY='your-langsmith-api-key'
export LANGCHAIN_PROJECT='my-project'

# Your LangChain code is now automatically traced
python my_app.py

# Open LangSmith dashboard to view traces
# https://smith.langchain.com

Example

Adding custom tracing to non-LangChain code:

from langsmith import traceable
import openai

client = openai.OpenAI()

@traceable(name='generate_summary')
def summarize(text: str) -> str:
    response = client.chat.completions.create(
        model='gpt-4',
        messages=[{'role': 'user', 'content': f'Summarize: {text}'}]
    )
    return response.choices[0].message.content

# Every call is now traced in LangSmith
result = summarize('Long article text here...')

Related on TokRepo

• AI gateway providers — Alternative LLM observability
• Multi-agent frameworks — LangGraph deep-dive

Common pitfalls

• LangSmith sends trace data to LangChain's servers. Ensure your security policy allows this for the data being processed.
• The free tier has trace retention limits. High-volume applications may need a paid plan for full trace history.
• Automatic tracing only works with LangChain. For other frameworks, use the @traceable decorator or manual trace API.
• Evaluation datasets need curation. Poor-quality test cases lead to misleading evaluation results.
• LangSmith is a separate service from LangChain the library. You need an account even if you already use LangChain.
• Review the official documentation before deploying to production to ensure compatibility with your specific environment and requirements.
• Start with default settings and customize incrementally. Changing too many configuration options at once makes debugging harder.