AutoGPTQ — Easy-to-Use GPTQ Quantization for Large Language Models

Introduction

AutoGPTQ implements the GPTQ post-training quantization algorithm with a developer-friendly API. It compresses pretrained language model weights from 16-bit to 4-bit or 3-bit integers using a one-shot calibration process, enabling models that previously required data center GPUs to run on consumer hardware.

What AutoGPTQ Does

• Quantizes pretrained LLM weights to 4-bit or 3-bit using the GPTQ algorithm
• Provides simple Python APIs for both quantization and inference
• Supports loading and running models quantized by the community from Hugging Face Hub
• Integrates with Hugging Face transformers, PEFT, and optimum for seamless workflows
• Uses optimized CUDA kernels for fast inference on quantized models

Architecture Overview

AutoGPTQ performs layer-by-layer quantization using a small calibration dataset. For each transformer layer, it computes the Hessian matrix of weight activations, then solves for optimal quantized weights that minimize reconstruction error. The quantized weights are stored in a custom format with group-wise scaling factors. At inference time, dedicated CUDA kernels dequantize weights on the fly during matrix multiplication, maintaining throughput while using a fraction of the memory.

Self-Hosting & Configuration

• Install via pip with CUDA support for GPU-accelerated quantization and inference
• Prepare a small calibration dataset (128-256 samples) representative of your use case
• Configure quantization parameters: bits (3 or 4), group_size, and description act order
• Save quantized models in a format compatible with Hugging Face Hub for sharing
• Load quantized models directly in transformers using the GPTQ backend

Key Features

• One-shot quantization requires only a small calibration set, no retraining needed
• 4-bit models use roughly 4x less VRAM than their 16-bit counterparts
• Marlin kernel integration provides fast int4 x fp16 matrix multiplication
• Direct compatibility with Hugging Face transformers for loading quantized models
• Group-wise quantization with configurable group sizes for accuracy-size tradeoffs

Comparison with Similar Tools

• QLoRA — quantizes for training with LoRA adapters; AutoGPTQ quantizes for efficient inference
• AWQ — activation-aware weight quantization; AutoGPTQ uses Hessian-based error minimization
• GGUF/llama.cpp — CPU-focused quantized inference; AutoGPTQ targets CUDA GPUs with optimized kernels
• GPTQModel — actively maintained fork of AutoGPTQ with additional model support and bug fixes

FAQ

Q: How much accuracy is lost with 4-bit quantization? A: GPTQ typically preserves over 99% of the original model quality on standard benchmarks when using 4-bit quantization with 128-group size.

Q: How long does quantization take? A: Quantizing a 7B model takes approximately 10-20 minutes on a modern GPU. Larger models scale roughly linearly with parameter count.

Q: Can I fine-tune a GPTQ-quantized model? A: Yes, GPTQ-quantized models work with QLoRA and PEFT for parameter-efficient fine-tuning on top of the compressed base weights.

Q: Is AutoGPTQ still maintained? A: The original repository is in maintenance mode. GPTQModel is the recommended actively developed successor for new projects.

Sources

• https://github.com/AutoGPTQ/AutoGPTQ
• https://huggingface.co/docs/transformers/quantization/gptq