BenchmarkDotNet — Precise .NET Performance Benchmarking

Introduction

BenchmarkDotNet is the standard .NET library for micro-benchmarking. It handles warmup iterations, statistical analysis, GC collection tracking, and result formatting so developers can measure performance accurately without building custom benchmarking infrastructure.

What BenchmarkDotNet Does

• Runs benchmarks with automatic warmup and multiple iterations
• Calculates mean, median, standard deviation, and confidence intervals
• Tracks memory allocations and garbage collection per benchmark
• Generates reports in Markdown, HTML, CSV, and other formats
• Supports parameterized benchmarks for comparing inputs

Architecture Overview

BenchmarkDotNet compiles each benchmark method into an isolated project, runs it in a separate process to avoid side effects, and collects timing data across many iterations. It uses statistical engines to determine when results are stable, applies outlier detection, and accounts for JIT compilation and GC pauses. The diagnoser infrastructure allows plugging in memory, threading, and hardware counter analyzers.

Self-Hosting & Configuration

• Install via NuGet: dotnet add package BenchmarkDotNet
• Run benchmarks in Release mode for accurate results
• Add [MemoryDiagnoser] to track allocations and GC collections
• Use [Params] attribute to sweep over multiple input values
• Configure custom job settings for different runtimes (.NET 6, 8, 9, NativeAOT)

Key Features

• Statistical engine with outlier detection and confidence intervals
• Cross-runtime comparison (CoreCLR, Mono, NativeAOT) in a single run
• Built-in diagnosers for memory, disassembly, and ETW events
• Baseline comparison with ratio columns and percentage diffs
• Export to GitHub-flavored Markdown for easy sharing

Comparison with Similar Tools

• dotnet-trace / dotnet-counters — runtime profiling tools; measure live apps, not isolated methods
• Stopwatch — manual timing; no warmup, no statistics, no allocation tracking
• NBench — .NET benchmark framework by Petabridge; less community adoption
• Benchmark.js (JavaScript) — similar concept for JS; different ecosystem
• Google Benchmark (C++) — micro-benchmarking for C++; similar philosophy, different language

FAQ

Q: Why must I run benchmarks in Release mode? A: Debug mode disables compiler optimizations and JIT inlining, producing results that do not reflect production performance.

Q: Can I benchmark async methods? A: Yes. Mark your benchmark method as returning Task and BenchmarkDotNet will await it correctly, measuring the full async execution.

Q: How do I compare two implementations? A: Mark one benchmark as [Benchmark(Baseline = true)] and BenchmarkDotNet will show ratio columns comparing all other benchmarks to the baseline.

Q: Does BenchmarkDotNet work with NativeAOT? A: Yes. Configure a Job targeting NativeAOT and BenchmarkDotNet will compile and run the benchmark using ahead-of-time compilation.

Sources

• https://github.com/dotnet/BenchmarkDotNet
• https://benchmarkdotnet.org/