Ragas — Evaluate RAG & LLM Applications

What it is

Ragas is a Python framework for evaluating Retrieval-Augmented Generation (RAG) pipelines and LLM applications. It provides objective metrics such as faithfulness, answer relevancy, context precision, and context recall. Ragas also generates synthetic test datasets so you can evaluate your RAG pipeline without manually curating question-answer pairs.

Ragas targets ML engineers and developers building production RAG systems who need quantitative evaluation beyond manual spot-checking. If you run a retrieval pipeline feeding context to an LLM, Ragas tells you how well the system performs across measurable dimensions.

How it saves time or tokens

Manually evaluating RAG outputs requires reading hundreds of responses and judging quality by hand. Ragas automates this with LLM-as-judge metrics that score each response on multiple axes. The synthetic test data generator creates diverse question-answer pairs from your documents, eliminating the hours spent crafting test sets. This workflow provides the pip install and evaluation script ready to run.

How to use

1. Install Ragas:

pip install ragas

1. Prepare your evaluation dataset with questions, ground truth answers, retrieved contexts, and LLM-generated answers.

1. Run the evaluation:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset

eval_dataset = Dataset.from_dict({
    'question': ['What is RAG?'],
    'answer': ['RAG combines retrieval with generation...'],
    'contexts': [['Retrieval-augmented generation...']],
    'ground_truth': ['RAG is a technique that...']
})

result = evaluate(eval_dataset, metrics=[faithfulness, answer_relevancy, context_precision])
print(result)

Example

# Generate synthetic test data from your documents
from ragas.testset import TestsetGenerator
from langchain_openai import ChatOpenAI

generator = TestsetGenerator.from_langchain(
    generator_llm=ChatOpenAI(model='gpt-4o'),
    critic_llm=ChatOpenAI(model='gpt-4o')
)

# Load your documents
from langchain_community.document_loaders import DirectoryLoader
loader = DirectoryLoader('./docs/', glob='**/*.md')
documents = loader.load()

testset = generator.generate_with_langchain_docs(
    documents, test_size=20
)
print(testset.to_pandas())

Related on TokRepo

• RAG tools -- Other retrieval-augmented generation frameworks and utilities
• Testing tools -- Automated testing and quality assurance for AI applications

Common pitfalls

• Running evaluation without setting an OpenAI API key. Ragas uses LLM-as-judge by default, which requires an LLM provider configured via environment variables.
• Using too few test samples leads to unreliable metric scores. Aim for at least 50 question-answer pairs for statistically meaningful results.
• Confusing context precision with context recall. Precision measures how much of the retrieved context is relevant; recall measures how much relevant context was retrieved.