# LM Evaluation Harness — Standardized LLM Benchmarking Framework

> The standard framework by EleutherAI for evaluating language models on hundreds of benchmarks, used by the Open LLM Leaderboard and research labs worldwide to produce reproducible evaluation results.

## Install

Save in your project root:

# LM Evaluation Harness — Standardized LLM Benchmarking Framework

## Quick Use
```bash
pip install lm-eval
```
```bash
lm_eval --model hf 
    --model_args pretrained=meta-llama/Llama-2-7b-hf 
    --tasks hellaswag,arc_challenge,mmlu 
    --batch_size 8
```

## Introduction
The LM Evaluation Harness is EleutherAI's framework for reproducible evaluation of language models across hundreds of academic benchmarks. It provides a unified interface to test any Hugging Face, OpenAI API, or custom model against standardized tasks, making it the backbone of the Open LLM Leaderboard and a standard tool in LLM research.

## What LM Evaluation Harness Does
- Evaluates language models on 400+ benchmarks including MMLU, HellaSwag, ARC, and GSM8K
- Supports multiple model backends: Hugging Face, OpenAI API, vLLM, GGUF, and custom implementations
- Handles few-shot prompting, chain-of-thought, and generation-based evaluation automatically
- Produces reproducible results with deterministic sampling and standardized prompt formats
- Enables custom task creation with a YAML-based configuration system

## Architecture Overview
The harness separates evaluation into three components: Models (how to query the LM), Tasks (what to evaluate), and Metrics (how to score). Models implement a common interface for log-likelihood and generation requests. Tasks define prompt templates, few-shot examples, and answer parsing. The evaluation loop batches requests efficiently and computes metrics across the full dataset with configurable aggregation.

## Self-Hosting & Configuration
- Install via pip with optional extras for specific model backends
- Run evaluations from the command line with model and task arguments
- Configure tasks via YAML files specifying prompt format, metrics, and few-shot count
- Use the Python API for programmatic evaluation in automated pipelines
- Parallelize across GPUs with accelerate or distribute via the vLLM backend

## Key Features
- 400+ built-in evaluation tasks covering reasoning, knowledge, math, coding, and more
- YAML-based task configuration for adding custom benchmarks without writing code
- Support for chat-templated evaluation matching model-specific prompt formats
- Caching and request deduplication for efficient repeated evaluations
- Group-based evaluation for multi-task benchmarks like MMLU (57 subjects)

## Comparison with Similar Tools
- **HELM (Stanford)** — comprehensive but heavier framework; lm-eval is lighter and more widely adopted for quick benchmarking
- **OpenCompass** — Chinese-focused evaluation suite; lm-eval covers primarily English benchmarks
- **Eleuther Harness vs Hugging Face evaluate** — evaluate provides individual metrics; lm-eval provides the full orchestration pipeline
- **BigBench** — Google's benchmark collection; lm-eval implements many BigBench tasks as evaluation targets
- **DeepEval** — focuses on LLM application testing; lm-eval targets model-level academic benchmarks

## FAQ
**Q: Is this what powers the Open LLM Leaderboard?**
A: Yes. Hugging Face's Open LLM Leaderboard uses this harness to produce all benchmark scores.

**Q: Can I evaluate API-based models like GPT-4?**
A: Yes. The harness supports OpenAI-compatible APIs and can evaluate any model accessible via HTTP endpoints.

**Q: How do I add a custom evaluation task?**
A: Create a YAML config file defining the dataset source, prompt template, answer parsing logic, and metric. No Python code required for standard formats.

**Q: Does it support multi-GPU evaluation?**
A: Yes. Use the accelerate integration or vLLM backend for distributed evaluation across multiple GPUs.

## Sources
- https://github.com/EleutherAI/lm-evaluation-harness
- https://docs.eleuther.ai/lm-eval/

---
Source: https://tokrepo.com/en/workflows/asset-a682e826
Author: AI Open Source