Celery — Distributed Task Queue for Python

What it is

Celery is the most widely used distributed task queue for Python. It handles background job processing, scheduled periodic tasks, and real-time message processing with support for multiple message brokers (RabbitMQ, Redis, Amazon SQS) and result backends (Redis, PostgreSQL, MongoDB).

It targets Python developers who need to offload time-consuming work (sending emails, processing images, running ML inference, calling external APIs) from their web request cycle into asynchronous background workers.

How it saves time or tokens

Without Celery, long-running operations block web requests, causing timeouts and poor user experience. Celery moves these operations to separate worker processes that execute independently. The web application returns immediately while the task runs in the background. Celery Beat handles periodic scheduling, replacing cron jobs with a more manageable, code-defined system.

How to use

1. Install Celery: pip install celery[redis] (with Redis as both broker and result backend).
2. Define tasks as decorated functions with @app.task.
3. Start a worker process with celery -A myapp worker and call tasks with .delay() or .apply_async().

Example

# tasks.py
from celery import Celery

app = Celery('myapp', broker='redis://localhost:6379/0',
             backend='redis://localhost:6379/1')

@app.task
def send_email(to, subject, body):
    # Simulate email sending
    import time
    time.sleep(5)
    return f'Email sent to {to}'

# In your web application:
result = send_email.delay('user@example.com', 'Hello', 'World')
print(result.id)  # Task ID for tracking
print(result.get(timeout=30))  # Wait for result

Related on TokRepo

• Automation tools — Task and workflow automation
• Coding tools — Python development frameworks

Common pitfalls

• Celery tasks must be serializable. Avoid passing database connections, file handles, or other non-serializable objects as task arguments.
• Memory leaks in long-running workers accumulate over time. Use the --max-tasks-per-child flag to restart workers after a set number of tasks.
• Celery Beat (the periodic task scheduler) should run as a single instance. Running multiple Beat processes causes duplicate task execution.