Faktory — Language-Agnostic Background Job Server

Introduction

Faktory is a centralized background job server created by Mike Perham (author of Sidekiq). While Sidekiq is Ruby-specific, Faktory externalizes the job queue into a standalone server that any language can connect to. Workers in Go, Python, Ruby, Rust, Node.js, or Elixir connect via a simple TCP protocol to fetch and execute jobs, making Faktory ideal for polyglot architectures.

What Faktory Does

• Accepts job payloads from producers via a TCP protocol and persists them in queues
• Dispatches jobs to workers in any language using client libraries and a simple RESP-like protocol
• Retries failed jobs with configurable exponential backoff and dead-job handling
• Supports scheduled jobs (run at a future time), unique jobs, and job batches with callbacks
• Provides a built-in web dashboard for monitoring queues, retries, scheduled jobs, and worker status

Architecture Overview

Faktory runs as a single-process server storing job data in an embedded RocksDB database. Producers push JSON job payloads over TCP. The server enqueues jobs by priority into named queues. Workers open persistent connections, issue FETCH commands to pull jobs, process them, and send ACK (success) or FAIL (retry). The server tracks in-flight jobs and requeues them if workers disconnect without acknowledgment. An HTTP server hosts the monitoring UI.

Self-Hosting & Configuration

• Deploy via Docker or download the binary; data persists in a RocksDB directory
• Configure via /etc/faktory/conf.d/*.toml for queues, concurrency limits, and retention policies
• Set up authentication with a password via the FAKTORY_PASSWORD environment variable
• Enable TLS for production deployments to encrypt the worker-server TCP protocol
• Use the web UI (port 7420) for real-time monitoring; protect with reverse proxy authentication

Key Features

• Language-agnostic protocol with official clients for Ruby, Go, Python, Rust, Node.js, and Elixir
• Job batches that group related jobs and fire a callback when all complete or any fail
• Unique jobs preventing duplicate enqueue of the same job within a configurable time window
• Queue weighting and strict priority ordering for fine-grained scheduling control
• Enterprise features (optional license) include periodic jobs, expiring jobs, and rate limiting

Comparison with Similar Tools

• Sidekiq — Ruby-only, in-process with Redis; Faktory is language-agnostic with its own persistence
• Celery — Python-focused with Redis/RabbitMQ broker; Faktory is a standalone server for any language
• BullMQ — Node.js job queue on Redis; Faktory provides a server with built-in persistence and web UI
• Temporal — Full workflow orchestration with history; Faktory is simpler for fire-and-forget background tasks
• RabbitMQ — General message broker; Faktory is purpose-built for background jobs with retries and scheduling

FAQ

Q: What happens if a worker crashes mid-job? A: Faktory tracks job reservations with timeouts. If a worker disconnects without ACK or FAIL, the job is automatically returned to the queue after the reservation expires (default 30 minutes).

Q: Can Faktory replace Redis for job queuing? A: Yes, for background job workloads. Faktory embeds its own storage (RocksDB) so you don't need a separate Redis instance. However, it's specialized for jobs, not general-purpose caching.

Q: How does Faktory handle job priorities? A: Workers fetch from queues in the order specified. By listing queues as ["critical", "default", "low"], workers always drain higher-priority queues first (strict ordering) or use weighted random.

Q: Is there a free version? A: Yes. The open-source Faktory includes queues, retries, scheduled jobs, web UI, and batches. Enterprise adds periodic jobs, expiring jobs, rate limiting, and queue throttling.

Sources

• https://github.com/contribsys/faktory
• https://contribsys.com/faktory/