Latitude — AI Agent Engineering Platform

What it is

Latitude is an open-source platform for building, evaluating, and monitoring AI agents in production. It provides observability dashboards, a prompt playground for iterating on prompts, LLM-as-judge evaluations for automated quality scoring, and experiment comparison for A/B testing different agent configurations.

Latitude targets engineering teams deploying AI agents who need visibility into agent behavior, quality metrics, and a systematic way to improve prompts.

How it saves time or tokens

Latitude's prompt playground lets you test prompt variations side-by-side without deploying. You iterate faster because you see results immediately, avoiding the deploy-test-revert cycle.

The LLM-as-judge evaluation automates quality assessment. Instead of manually reviewing agent outputs, Latitude scores them against criteria you define, catching regressions early.

How to use

1. Deploy Latitude with Docker: follow the self-hosting guide in the repository
2. Connect your LLM providers (OpenAI, Anthropic, etc.)
3. Create prompts in the playground and test with different inputs
4. Set up evaluations to automatically score agent outputs

Example

// Latitude SDK: run a prompt and evaluate the result
import { Latitude } from '@latitude-data/sdk';

const latitude = new Latitude('your-api-key');

// Run a prompt
const result = await latitude.prompts.run('customer-support-agent', {
  parameters: {
    customer_query: 'How do I reset my password?',
    context: 'User has a Pro account created in 2024'
  }
});

// Log the result for evaluation
await latitude.logs.create({
  prompt: 'customer-support-agent',
  response: result.text,
  metadata: { category: 'account-management' }
});

Related on TokRepo

• Monitoring tools -- AI observability and monitoring
• AI agent tools -- Agent frameworks and platforms

Common pitfalls

• Self-hosting requires PostgreSQL and Redis; ensure these are provisioned before deploying Latitude
• LLM-as-judge evaluations consume additional tokens; budget for evaluation costs alongside agent inference costs
• Prompt versioning in Latitude is separate from Git; establish a workflow to keep both in sync