Beanstalkd — Simple Fast Work Queue

Introduction

Beanstalkd is a simple, fast, general-purpose work queue originally designed to reduce page latency in high-volume web applications by running time-consuming tasks asynchronously. Written in C, it provides a lightweight protocol over TCP with tubes (named queues), priority ordering, and delayed jobs.

What Beanstalkd Does

• Provides named tubes for logically separating job queues across different task types
• Supports job priorities so urgent work is processed before background tasks
• Offers delayed jobs that become available after a specified number of seconds
• Implements time-to-run (TTR) with automatic job release if a worker dies mid-processing
• Persists jobs to a binlog on disk for crash recovery

Architecture Overview

Beanstalkd is a single-threaded, event-driven server written in C using an internal event loop. Jobs are stored in an in-memory priority queue backed by a min-heap, giving O(log n) put and reserve operations. Each tube maintains its own ready, delayed, and buried queues. The text-based protocol runs over TCP, making it easy to implement clients in any language. Optional binlog persistence writes jobs to an append-only file for durability.

Self-Hosting & Configuration

• Single binary with zero dependencies; compile from source or install via package manager
• Configure listen address and port with -l and -p flags
• Enable persistence with -b /path/to/binlog for crash recovery
• Set max job size with -z (default 65535 bytes)
• Monitor with stats and stats-tube commands over the protocol

Key Features

• Sub-millisecond latency for put and reserve operations
• Minimal memory footprint suitable for embedded and resource-constrained environments
• Client libraries available in 20+ languages including Python, Ruby, PHP, Go, and Java
• Buried job state for manual inspection of failed jobs without losing them
• Kick command to move buried or delayed jobs back to the ready queue

Comparison with Similar Tools

• Redis (with lists) — More versatile but heavier; Beanstalkd is purpose-built for job queues with priorities and TTR
• RabbitMQ — Full AMQP broker with clustering and routing; Beanstalkd is simpler with lower overhead
• Amazon SQS — Managed cloud service; Beanstalkd is self-hosted with no external dependencies
• Celery — Python task framework that can use Beanstalkd as a broker; Beanstalkd is the transport layer

FAQ

Q: Does Beanstalkd support clustering or replication? A: No, Beanstalkd is a single-server daemon. For high availability, run multiple instances behind application-level routing.

Q: What happens if Beanstalkd crashes? A: With binlog enabled, jobs are recovered on restart. Without it, in-memory jobs are lost.

Q: Is there a web UI for monitoring? A: Community tools like Beanstalk Console and Aurora provide web-based monitoring interfaces.

Q: How does TTR work? A: When a worker reserves a job, it has TTR seconds to delete or release it. If TTR expires, the job is automatically released back to the ready queue for another worker.

Sources

• https://github.com/beanstalkd/beanstalkd
• https://beanstalkd.github.io/