Evidently — ML & LLM Monitoring with 100+ Metrics

What it is

Evidently is an open-source Python library for monitoring machine learning models and LLM applications. It provides over 100 built-in metrics covering data drift, model quality, classification and regression performance, and text analysis. You can generate reports as interactive HTML dashboards, run tests as part of CI/CD pipelines, and monitor production models in real-time.

It targets ML engineers and data scientists who need to track model performance after deployment and catch degradation early.

How it saves time or tokens

Evidently automates the monitoring work that teams often do manually with custom scripts. Instead of writing drift detection, quality metrics, and visualization code, you configure a preset and get a comprehensive report. For LLM applications, the text metrics analyze output quality (length, sentiment, toxicity, patterns) without building custom evaluation pipelines. The test suite integration catches regressions in CI before they reach production.

How to use

1. Install the library:

pip install evidently

1. Generate a data drift report:

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
import pandas as pd

reference = pd.read_csv('training_data.csv')
current = pd.read_csv('production_data.csv')

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=reference, current_data=current)
report.save_html('drift_report.html')

1. Run automated tests:

from evidently.test_suite import TestSuite
from evidently.test_preset import DataDriftTestPreset

suite = TestSuite(tests=[DataDriftTestPreset()])
suite.run(reference_data=reference, current_data=current)
print(suite)  # Pass/Fail for each test

Example

from evidently.report import Report
from evidently.metric_preset import TextEvals
from evidently import ColumnMapping
import pandas as pd

# Monitor LLM output quality
llm_outputs = pd.DataFrame({
    'prompt': ['Summarize this article', 'Write a haiku', 'Explain recursion'],
    'response': ['The article discusses...', 'Autumn leaves falling...', 'Recursion is when...']
})

column_mapping = ColumnMapping(
    text_features=['response']
)

report = Report(metrics=[TextEvals(column_name='response')])
report.run(current_data=llm_outputs, column_mapping=column_mapping)
report.save_html('llm_quality_report.html')

Related on TokRepo

• AI tools for monitoring -- ML and LLM monitoring platforms
• AI tools for testing -- Automated testing and evaluation tools

Common pitfalls

• Data drift detection requires a reference dataset. Without a good reference (typically your training data), drift alerts may be noisy or misleading.
• The 100+ metrics can be overwhelming. Start with presets (DataDriftPreset, DataQualityPreset) and add individual metrics only when you need specific insights.
• Interactive HTML reports grow large for big datasets. For production monitoring, use the Evidently monitoring UI or export metrics to Grafana instead of HTML files.