Apache RocketMQ — Cloud-Native Messaging and Streaming Platform

Introduction

Apache RocketMQ is a distributed messaging and streaming platform originally developed at Alibaba and donated to the Apache Software Foundation. It handles trillion-level message routing with low latency and high reliability, supporting both messaging and event streaming patterns for microservice architectures, data pipelines, and IoT workloads.

What RocketMQ Does

• Delivers messages with multiple patterns: pub/sub, point-to-point, request-reply, and ordered messaging
• Provides transactional messages for reliable event publishing tied to local database transactions
• Supports scheduled and delayed message delivery with configurable delay levels
• Offers message tracing, dead letter queues, and automatic retry for failed consumption
• Enables event streaming with lightweight topic-based partitioning and consumer groups

Architecture Overview

RocketMQ consists of four components. NameServer is a lightweight registry for broker routing. Brokers store and deliver messages, organized in master-slave or DLedger (Raft) clusters for high availability. Producers publish messages to topics via NameServer-resolved broker addresses. Consumers pull or are pushed messages within consumer groups with rebalancing. In RocketMQ 5.x, a Proxy layer adds gRPC support and decouples compute from storage.

Self-Hosting & Configuration

• Deploy NameServer instances (at least 2 for HA) and configure broker.conf with NameServer addresses
• Start brokers in cluster mode with master-slave or DLedger replication for durability
• Configure memory, disk flush policy, and message retention in broker configuration
• Deploy the RocketMQ Dashboard (web UI) for topic management, consumer monitoring, and message queries
• Use Docker Compose or Kubernetes Operator for containerized deployment

Key Features

• Transactional messages with half-message mechanism for exactly-once semantics with databases
• Ordered message delivery guaranteed within a single queue (partition)
• Message filtering by tags and SQL92 expressions on message properties
• DLedger-based Raft replication for automatic leader election without external coordination
• Multi-language clients including Java, Go, C++, Python, Rust, and Node.js

Comparison with Similar Tools

• Apache Kafka — Log-based streaming with higher throughput for analytics; RocketMQ adds transactional messages, delayed delivery, and simpler operations
• Apache Pulsar — Multi-tenant with tiered storage; RocketMQ is simpler to operate with built-in transaction support
• RabbitMQ — AMQP broker for complex routing; RocketMQ scales better horizontally for high-throughput messaging
• NSQ — Decentralized Go messaging; RocketMQ provides stronger durability guarantees and richer features
• NATS — Ultra-lightweight pub/sub; RocketMQ adds persistence, transactions, and scheduled delivery

FAQ

Q: How does RocketMQ compare to Kafka for throughput? A: Both handle millions of messages per second. Kafka excels at sequential log streaming; RocketMQ performs better for diverse messaging patterns like transactions and delays.

Q: What are transactional messages in RocketMQ? A: RocketMQ sends a half-message (invisible to consumers), executes the local transaction, then commits or rolls back the message based on the transaction outcome.

Q: Can RocketMQ run in Kubernetes? A: Yes. The RocketMQ Operator provides custom resources for deploying and managing NameServer, Broker, and Proxy on Kubernetes.

Q: Does RocketMQ support message replay? A: Yes. Consumers can reset their offset to any timestamp to replay historical messages within the retention period.

Sources

• https://github.com/apache/rocketmq
• https://rocketmq.apache.org/docs/