RQ — Simple Python Job Queue Backed by Redis

Introduction

RQ (Redis Queue) is a Python library for queueing and processing background jobs. It uses Redis as a message broker and is designed to be the simplest possible task queue with no boilerplate, no configuration files, and no complex dependencies beyond Redis itself.

What RQ Does

• Enqueues Python function calls as background jobs stored in Redis lists
• Runs workers that pull jobs from named queues and execute them in forked processes
• Tracks job status, results, and failures with automatic retry support
• Provides a built-in dashboard (rq-dashboard) for monitoring queues and workers
• Supports job dependencies, timeouts, TTL, and scheduled execution

Architecture Overview

RQ serializes Python function references and arguments using pickle and pushes them onto Redis lists. Workers run as separate processes, blocking on Redis BLPOP to receive jobs. Each job executes in a forked child process for isolation, so a crashed job does not take down the worker. Results are stored back in Redis with configurable TTL, and failed jobs are moved to a dedicated failed queue for inspection or retry.

Self-Hosting & Configuration

• Install: pip install rq (requires a running Redis instance)
• Start a worker: rq worker [queue_name] from the command line
• Configure Redis connection via environment variable RQ_REDIS_URL or pass a Redis object in code
• Install the dashboard: pip install rq-dashboard then rq-dashboard to launch the web UI
• Deploy workers with systemd, supervisor, or container orchestrators for production use

Key Features

• Minimal API: enqueue any Python callable with queue.enqueue(func, *args, **kwargs)
• Named queues with priority ordering so workers process high-priority work first
• Job dependencies: chain jobs so one starts only after its prerequisite succeeds
• Automatic retries with configurable max attempts and exponential backoff
• Lightweight footprint with no JVM, no Erlang, and no external config files

Comparison with Similar Tools

• Celery — full-featured distributed task queue with routing, canvas workflows, and multiple broker backends; RQ trades features for simplicity
• Dramatiq — offers reliability features like middleware and rate limiting; RQ is simpler but less opinionated about error handling
• Huey — similar lightweight queue supporting Redis and SQLite; RQ has a larger community and more third-party integrations
• ARQ — async-native Redis queue using asyncio; RQ uses synchronous fork-based execution
• Taskiq — modern async task framework with multiple brokers; RQ focuses exclusively on Redis with a synchronous model

FAQ

Q: Can RQ handle millions of jobs? A: Yes, Redis handles large queues efficiently. Scale horizontally by running more workers across multiple machines pointing at the same Redis instance.

Q: Does RQ support periodic or cron-like tasks? A: Not natively. Use rq-scheduler or an external cron to enqueue jobs on a schedule.

Q: What happens when a worker crashes mid-job? A: The job is moved to the failed queue. RQ does not re-enqueue it automatically unless you configure retries or use a worker health-check process.

Q: Can I use RQ with async/await code? A: RQ workers execute jobs synchronously in forked processes. For async workloads, consider ARQ or wrap async calls with asyncio.run() inside the job function.

Sources

• https://github.com/rq/rq
• https://python-rq.org/