Machinery — Distributed Task Queue for Go

Introduction

Machinery is a distributed task queue library for Go that provides an asynchronous work distribution system similar to Python's Celery. It lets developers define tasks as Go functions, send them to a broker (Redis, AMQP, or SQS), and process them with worker pools. Machinery handles retries, result storage, task chaining, and scheduled execution.

What Machinery Does

• Distributes tasks across multiple workers using Redis, RabbitMQ (AMQP), or AWS SQS as the message broker
• Stores task results in Redis, Memcached, MongoDB, or a custom backend for later retrieval
• Supports task chaining (sequential execution), grouping (parallel execution), and chords (fan-out then aggregate)
• Provides configurable retry logic with exponential backoff for failed tasks
• Schedules tasks for future execution using ETA timestamps

Architecture Overview

Machinery follows a broker-worker-backend architecture. The server component registers task functions and sends task signatures to the broker. Workers consume messages from broker queues, execute the matching function, and store results in the result backend. Task signatures describe the function name, arguments, and execution metadata (retries, ETA, routing key). The broker abstraction supports Redis lists, AMQP queues, and SQS queues behind a common interface. Workers run in goroutine pools with configurable concurrency, and the result backend provides blocking Get calls for synchronous result retrieval.

Self-Hosting & Configuration

• Install via go get and configure the broker URL, result backend URL, and default queue name in a config struct or YAML file
• Register task functions with server.RegisterTask before starting workers
• Start workers with worker.Launch() specifying concurrency level and queue bindings
• Deploy multiple worker processes across machines for horizontal scaling; each connects to the same broker
• Monitor task execution via the result backend or integrate with logging and metrics systems

Key Features

• Multiple broker backends (Redis, AMQP, SQS) with a unified API for portability
• Task workflows: chains for sequential steps, groups for parallel execution, and chords for fan-out/fan-in patterns
• Configurable retry with exponential backoff, max retries, and custom retry intervals
• Delayed task execution via ETA for scheduling future work
• Strongly typed task arguments with automatic serialization and deserialization

Comparison with Similar Tools

• Celery (Python) — the inspiration for Machinery; Celery offers a more mature ecosystem while Machinery provides a native Go solution
• Asynq — newer Go task queue focused on Redis; simpler API but fewer workflow patterns (no groups/chords)
• Faktory — language-agnostic worker server; Machinery is a pure Go library with no external server process
• Temporal — workflow orchestration platform with durable execution; Machinery is lighter-weight for simpler task distribution
• Bull/BullMQ (Node.js) — Redis-based queue for JavaScript; Machinery serves the Go ecosystem with similar capabilities

FAQ

Q: Which broker should I choose? A: Redis is the simplest to set up and works well for most workloads. Use AMQP (RabbitMQ) for advanced routing and guaranteed delivery, or SQS for AWS-native deployments.

Q: How do I scale workers? A: Run multiple worker processes on different machines, each connecting to the same broker. Increase the concurrency parameter to run more goroutines per process.

Q: Can I prioritize certain tasks? A: Yes. Use different queues with different workers to create priority lanes, or use AMQP priority queues for message-level prioritization.

Q: Is Machinery production-ready? A: Yes. Machinery has been used in production Go services for task distribution, email sending, data processing, and webhook delivery.

Sources

• https://github.com/RichardKnop/machinery
• https://pkg.go.dev/github.com/RichardKnop/machinery/v2