pgmq — Lightweight Message Queue Built on PostgreSQL

Introduction

pgmq is a lightweight message queue implemented as a PostgreSQL extension. It brings message queue semantics — send, read with visibility timeout, delete, and archive — directly into PostgreSQL without requiring a separate broker. Applications already using PostgreSQL can add job queues and task processing without additional infrastructure.

What pgmq Does

• Provides send, read, delete, and archive operations on named queues
• Implements visibility timeouts so messages are hidden from other consumers while being processed
• Supports batch operations for sending and reading multiple messages at once
• Archives processed messages to a separate table for auditing
• Offers both SQL function calls and client libraries for Python, Rust, and other languages

Architecture Overview

Each queue is backed by two PostgreSQL tables — one for active messages and one for archived messages. The read function uses SELECT ... FOR UPDATE SKIP LOCKED for safe concurrent access. Visibility timeouts are enforced by setting a future timestamp on read; messages become visible again if not deleted before timeout. All operations run within PostgreSQL transactions, leveraging existing WAL replication and backup.

Self-Hosting & Configuration

• Install as a PostgreSQL extension via CREATE EXTENSION pgmq (available on Trunk, pgxn, or from source)
• Works with PostgreSQL 12 and above
• No separate daemon or broker process required
• Queues are created and managed with SQL function calls
• Inherits PostgreSQL's replication, backup, and high-availability configurations

Key Features

• Visibility timeout prevents duplicate processing without external coordination
• SKIP LOCKED based reads for high-concurrency consumer patterns
• Message archival for audit trails and reprocessing
• Batch send and read operations for throughput optimization
• Partitioned queue tables available for high-volume scenarios

Comparison with Similar Tools

• RabbitMQ — Full-featured broker with protocols like AMQP; pgmq is simpler with no extra infrastructure
• Redis Streams — In-memory with persistence options; pgmq uses PostgreSQL's durable storage natively
• Amazon SQS — Managed cloud queue; pgmq is self-hosted and avoids vendor lock-in
• Celery — Task framework needing a broker; pgmq can serve as that broker within PostgreSQL

FAQ

Q: Does pgmq guarantee exactly-once delivery? A: It provides at-least-once delivery via visibility timeouts. Exactly-once processing depends on consumers deleting messages after successful handling.

Q: How does pgmq handle failed consumers? A: If a consumer crashes before deleting a message, the visibility timeout expires and the message becomes available for another consumer to read.

Q: Can pgmq handle high throughput? A: It performs well for moderate workloads. For very high message rates, a dedicated broker may be more appropriate, but partitioned queues help scale within PostgreSQL.

Sources

• https://github.com/tembo-io/pgmq
• https://tembo.io/pgmq