Asynq — Simple Distributed Task Queue for Go

Introduction

Asynq is a Go library for building reliable background task processing systems. It uses Redis as its message broker and provides features like task scheduling, automatic retries, priority queues, and a monitoring dashboard, all with a minimal and idiomatic Go API.

What Asynq Does

• Enqueues tasks as serialized payloads into Redis and processes them asynchronously in worker goroutines
• Supports delayed tasks, periodic (cron-style) scheduling, and unique task deduplication
• Automatically retries failed tasks with configurable backoff and maximum retry counts
• Assigns tasks to priority queues so critical jobs run before lower-priority ones
• Ships with Asynqmon, a web UI and CLI for inspecting queues, tasks, and worker status

Architecture Overview

Asynq consists of a client that enqueues tasks and a server (worker) that dequeues and processes them. Both communicate through Redis. The server runs a configurable pool of worker goroutines that pull tasks from sorted sets in Redis. Task state transitions (pending, active, retry, completed, archived) are managed atomically via Lua scripts executed inside Redis, ensuring consistency without external coordination.

Self-Hosting & Configuration

• Requires a running Redis 6.2+ instance; configure the connection via asynq.RedisClientOpt
• Define task handlers by implementing the asynq.Handler interface with a ProcessTask method
• Set concurrency, queue priorities, and retry policies when creating the asynq.Server
• Deploy Asynqmon as a standalone binary or Docker container for the monitoring dashboard
• Use asynq.Scheduler to register periodic tasks with cron expressions at server startup

Key Features

• Minimal Go API: enqueue a task in three lines of code
• Redis-backed with atomic Lua scripts for reliable task state management
• Built-in cron scheduler for recurring jobs without an external cron daemon
• Unique task constraints prevent duplicate enqueues within a configurable TTL
• Asynqmon web UI provides real-time visibility into queues, tasks, and workers

Comparison with Similar Tools

• Celery — Python-based, feature-rich but heavyweight; Asynq is Go-native and lighter
• Sidekiq — Ruby background jobs on Redis; similar Redis-backed design, different language ecosystem
• BullMQ — Node.js/TypeScript queue on Redis; comparable feature set for the JavaScript world
• Machinery — Go task queue supporting multiple brokers, but more complex configuration
• Faktory — language-agnostic job server; Asynq is simpler for Go-only stacks

FAQ

Q: Can Asynq run on Redis Cluster? A: Yes. Asynq supports Redis Cluster and Redis Sentinel for high availability setups.

Q: How does Asynq handle worker crashes? A: Active tasks have a lease timeout. If the worker does not complete or extend the lease, the task is automatically returned to the retry queue.

Q: Is there a maximum task payload size? A: Payloads are limited by Redis string size (512 MB), but in practice keeping payloads small (a few KB) and storing large data externally is recommended.

Q: Can I use Asynq without the web UI? A: Yes. Asynqmon is optional. The core library and CLI work independently.

Sources

• https://github.com/hibiken/asynq
• https://github.com/hibiken/asynqmon