DeepEval — LLM Testing Framework with 30+ Metrics

What it is

DeepEval is an open-source testing framework designed specifically for LLM applications. It works like pytest but adds 30+ evaluation metrics tailored to AI outputs, including answer relevancy, faithfulness, contextual precision, hallucination detection, and task completion scoring.

The framework targets ML engineers and backend developers building RAG pipelines, AI agents, or any application that needs automated quality checks on LLM outputs.

How it saves time or tokens

Manual evaluation of LLM outputs is slow and inconsistent. DeepEval automates the process with quantitative metrics, catching regressions in CI/CD before they reach production. All evaluations run locally on your machine, so no data leaves your environment and you avoid paying for external evaluation APIs.

How to use

1. Install DeepEval via pip: pip install -U deepeval.
2. Create a test file with test cases defining input, expected output, and retrieval context.
3. Run tests with deepeval test run test_llm.py -- results show pass/fail per metric.

Example

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

def test_rag_pipeline():
    test_case = LLMTestCase(
        input='What is DeepEval?',
        actual_output='DeepEval is an LLM testing framework.',
        retrieval_context=['DeepEval provides 30+ metrics for LLM evaluation.']
    )
    relevancy = AnswerRelevancyMetric(threshold=0.7)
    faithfulness = FaithfulnessMetric(threshold=0.8)
    assert_test(test_case, [relevancy, faithfulness])

Related on TokRepo

• AI Tools for Testing -- compare AI-powered testing and evaluation tools
• AI Tools for RAG -- explore retrieval-augmented generation frameworks and engines

Common pitfalls

• Setting metric thresholds too high initially causes false failures. Start with 0.5-0.7 and tighten as your pipeline matures.
• DeepEval strips types but does not validate LLM logic. Pair it with unit tests for deterministic code paths.
• The retrieval_context field is required for RAG metrics like faithfulness. Omitting it silently skips those checks.