# Sidekiq — Efficient Background Processing for Ruby

> Sidekiq is a background job framework for Ruby that uses threads to process many jobs concurrently within a single process. It is backed by Redis and handles millions of jobs per day in production.

## Install

Save in your project root:

# Sidekiq — Efficient Background Processing for Ruby

## Quick Use
```ruby
# Gemfile
gem 'sidekiq'
```
```ruby
# app/workers/hard_worker.rb
class HardWorker
  include Sidekiq::Job
  def perform(name, count)
    puts "Processing #{name} #{count} times"
  end
end
HardWorker.perform_async('task', 5)
```
```bash
bundle exec sidekiq
```

## Introduction
Sidekiq is a Ruby background job processor that uses threads instead of processes to handle concurrent work. A single Sidekiq process can handle dozens of jobs simultaneously, making it far more memory-efficient than alternatives that fork a new process per job.

## What Sidekiq Does
- Processes background jobs concurrently using a thread pool within a single Ruby process
- Stores job payloads in Redis for fast enqueue and dequeue operations
- Retries failed jobs automatically with exponential backoff
- Provides a real-time web dashboard for monitoring queues, retries, and dead jobs
- Supports scheduled jobs, batches, and rate limiting in its Pro and Enterprise tiers

## Architecture Overview
Sidekiq runs a multi-threaded Ruby process that polls Redis for job payloads. When a job is enqueued, it is serialized as JSON and pushed to a Redis list. The Sidekiq server pops jobs from queues, deserializes them, and executes the perform method in a thread from its pool. Failed jobs are moved to a retry set with configurable backoff. The web UI reads queue and retry data directly from Redis.

## Self-Hosting & Configuration
- Add the `sidekiq` gem to your Gemfile and run `bundle install`
- Configure Redis connection via `config/initializers/sidekiq.rb` or `REDIS_URL`
- Set concurrency (thread count) in `config/sidekiq.yml` based on available memory
- Mount the web dashboard in your Rails routes for monitoring
- Deploy Sidekiq as a separate process alongside your web server using systemd, Docker, or a process manager

## Key Features
- Thread-based concurrency that uses a fraction of the memory of process-based workers
- Automatic retry with exponential backoff and dead job tracking
- Built-in web UI with real-time queue depth, job latency, and error graphs
- Middleware pipeline for logging, error reporting, and custom job wrapping
- Battle-tested at scale, processing millions of jobs per day at companies worldwide

## Comparison with Similar Tools
- **Resque** — process-based Ruby job system; Sidekiq uses threads for better memory efficiency
- **Delayed::Job** — stores jobs in SQL; Sidekiq uses Redis for lower latency
- **GoodJob** — PostgreSQL-based; avoids Redis dependency but trades throughput
- **Celery** — Python job queue; Sidekiq serves the Ruby ecosystem with similar concepts
- **Faktory** — language-agnostic job server by the Sidekiq author; Sidekiq is Ruby-native

## FAQ
**Q: Does Sidekiq require Redis?**
A: Yes. Redis is the backing store for all job queues, scheduled sets, and retry data.

**Q: How many threads should I run?**
A: Start with 10 (the default). Increase if your jobs are I/O-bound; decrease if they are CPU-bound or memory-heavy.

**Q: Is thread safety a concern?**
A: Yes. Your job code must be thread-safe. Avoid shared mutable state and use connection pools for databases and external services.

**Q: What is the difference between Sidekiq OSS and Pro?**
A: The open-source version covers queues, retries, and the web UI. Pro adds batches, rate limiting, and reliability features.

## Sources
- https://github.com/sidekiq/sidekiq
- https://sidekiq.org

---
Source: https://tokrepo.com/en/workflows/89e00745-44f8-11f1-9bc6-00163e2b0d79
Author: AI Open Source