whichllm — Find the Best Local LLM for Your Hardware

Introduction

whichllm is a command-line tool that identifies the best-performing local LLM for your specific hardware. Instead of guessing which model fits your VRAM, it scans your system, checks available memory, and ranks models based on real benchmark data.

What whichllm Does

• Detects GPU type, VRAM capacity, and system RAM automatically
• Ranks compatible models by inference speed on similar hardware
• Filters models by quantization level, parameter count, and task type
• Runs local benchmarks to measure actual tokens-per-second throughput
• Recommends the highest-quality model that fits your available memory

Architecture Overview

whichllm queries your system for GPU and memory information, then cross-references a curated database of model benchmarks collected from community hardware profiles. It calculates which models fit in your available VRAM at various quantization levels and ranks them by quality-adjusted throughput. The optional bench command runs inference locally to measure real performance.

Self-Hosting & Configuration

• Install via pip: pip install whichllm
• No configuration file required; works out of the box
• Supports NVIDIA, AMD, and Apple Silicon GPUs
• Benchmark data is fetched from the project's community database
• Offline mode available with cached benchmark data

Key Features

• Hardware-aware model recommendations instead of trial-and-error
• VRAM-fit calculation accounting for quantization and context length
• Community-sourced benchmark data across hundreds of GPU configurations
• Side-by-side comparison of multiple models on your hardware profile
• Integration with Ollama and llama.cpp for direct model downloads

Comparison with Similar Tools

• Ollama — serves models but does not recommend which one fits; whichllm fills the selection gap
• LM Studio — GUI-based model browser; whichllm is CLI-first with hardware-aware ranking
• GPT4All — bundles selected models; whichllm covers a broader model catalog with benchmarks
• llama.cpp — inference engine; whichllm helps choose which model to run on it

FAQ

Q: Does it download models automatically? A: No. whichllm recommends models and provides download commands. You choose which to install.

Q: Which GPUs are supported? A: NVIDIA GPUs via CUDA, AMD GPUs via ROCm, and Apple Silicon via Metal are all detected.

Q: How accurate are the benchmark rankings? A: Rankings are based on community-reported data from matching hardware. Your actual results may vary slightly.

Q: Can I contribute my own benchmarks? A: Yes. Running whichllm bench generates results you can submit to the community database.

Sources

• https://github.com/Andyyyy64/whichllm
• https://pypi.org/project/whichllm/