Huey — Lightweight Python Task Queue with Minimal Dependencies

Introduction

Huey is a Python task queue that focuses on simplicity. Where Celery requires learning brokers, result backends, and serializers, Huey gives you background tasks, periodic scheduling, and result storage with a single decorator and minimal setup. It supports Redis, SQLite, and in-memory backends.

What Huey Does

• Enqueues Python functions as background tasks via a @task() decorator
• Schedules periodic jobs with crontab syntax using @periodic_task()
• Stores task results for later retrieval with configurable TTL
• Supports task retries with exponential backoff on failure
• Provides task pipelines for chaining dependent operations

Architecture Overview

Huey uses a producer-consumer model. The application enqueues serialized task calls into a storage backend (Redis, SQLite, or in-memory). The huey_consumer process polls the queue, deserializes tasks, and executes them in worker threads or greenlets. Results are written back to the same storage backend. The entire system runs with a single consumer process—no separate broker or result backend services needed for SQLite or in-memory modes.

Installation & Configuration

• Install via pip: pip install huey (Redis support included, no extras needed)
• Use RedisHuey() for production or SqliteHuey() for single-server setups
• Configure workers, threads, and scheduler in huey_consumer.py via CLI flags
• Integrates with Django via django-huey or Flask with minimal glue code
• Set immediate=True during development to run tasks synchronously without a consumer

Key Features

• Single-file simplicity: the entire library is well under 5,000 lines of code
• Multiple storage backends: Redis for distributed, SQLite for embedded, in-memory for testing
• Crontab-style periodic tasks without a separate scheduler process
• Task locking to prevent concurrent execution of the same task
• Pre- and post-execute hooks for logging, metrics, and error handling

Comparison with Similar Tools

• Celery — feature-rich but complex; Huey trades advanced routing for ease of setup
• RQ (Redis Queue) — Redis-only and simpler than Celery; Huey adds periodic tasks and SQLite support
• Dramatiq — actor-based with middleware; Huey is simpler with fewer abstractions
• APScheduler — scheduling library without a task queue; Huey combines both
• Arq — async task queue for Python; Huey uses threads and supports synchronous code natively

FAQ

Q: Can Huey replace Celery in production? A: For moderate workloads on a single server or small cluster, yes. Celery's advantage is advanced routing and multi-broker support at scale.

Q: Does Huey support async/await? A: Huey uses threads, not asyncio. Tasks are regular synchronous functions. For async workloads, consider Arq.

Q: How does the SQLite backend work in production? A: It stores tasks in a local SQLite file, suitable for single-server deployments. It avoids the need for a Redis instance but does not support distributed workers.

Q: Can I monitor task status? A: Yes. result.get() retrieves the return value, and result.is_complete() checks status. Huey also supports signal-based hooks for custom monitoring.

Sources

• https://github.com/coleifer/huey
• https://huey.readthedocs.io/