ExLlamaV2 — Fast Quantized LLM Inference

What it is

ExLlamaV2 is an inference engine for running quantized large language models on consumer NVIDIA GPUs. It uses optimized CUDA kernels to deliver high throughput with formats including EXL2, GPTQ, and HQQ. Features include PagedAttention and speculative decoding for additional speed.

ExLlamaV2 targets hobbyists and developers who want to run 7B-70B parameter models on hardware with 8-24GB VRAM. It extracts maximum performance from quantized models that other frameworks leave on the table.

The project is actively maintained and suitable for both individual developers and teams looking to integrate it into their existing toolchain. Documentation and community support are available for onboarding.

How it saves time or tokens

ExLlamaV2's custom CUDA kernels process quantized weights faster than general-purpose frameworks. The EXL2 format allows per-layer quantization (mixing 2-bit and 6-bit) to balance quality and memory. PagedAttention reduces memory waste from KV cache, letting you run longer contexts. Speculative decoding speeds up generation by 1.5-2x for compatible models.

How to use

1. Install ExLlamaV2 from pip or build from source with CUDA support.
2. Download a quantized model in EXL2 or GPTQ format from Hugging Face.
3. Load the model and run inference using the Python API or the built-in chat server.
4. Configure batch size, context length, and speculative decoding based on your GPU VRAM.

Example

from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Cache
from exllamav2.generator import ExLlamaV2StreamingGenerator

# Load a quantized model
config = ExLlamaV2Config('models/Llama-3-8B-EXL2-4.0bpw/')
model = ExLlamaV2(config)
cache = ExLlamaV2Cache(model, max_seq_len=4096)
model.load_autosplit(cache)

# Stream tokens
generator = ExLlamaV2StreamingGenerator(model, cache, model.tokenizer)
generator.set_stop_conditions([model.tokenizer.eos_token_id])

for token in generator.generate_simple('Explain transformers in 3 sentences.'):
    print(token, end='', flush=True)

Related on TokRepo

• Local LLM Runners — Compare ExLlamaV2 with other local inference engines.
• AI Tools for Coding — Use locally-run models for AI-assisted coding.

Common pitfalls

• Choosing a model too large for your VRAM. A 70B model at 4-bit quantization needs about 35GB VRAM. Check the model card for VRAM requirements before downloading.
• Using GPTQ when EXL2 is available. EXL2 offers better quality-per-bit through per-layer quantization. Prefer EXL2 models when they exist for your target model.
• Not setting max_seq_len appropriately. Higher context lengths consume more VRAM for KV cache. Set it to the maximum you actually need, not the model's theoretical maximum.
• Not reading the changelog before upgrading. Breaking changes between versions can cause unexpected failures in production. Pin your version and review release notes.