Hazelcast — Real-Time Distributed Computing Platform

Introduction

Hazelcast is a unified real-time data platform that combines distributed in-memory storage with stream processing and compute capabilities. Originally created as a distributed caching layer for Java applications, it has evolved into a general-purpose platform for building low-latency, high-throughput applications that need to process and act on data in real time across a cluster of machines.

What Hazelcast Does

• Provides distributed data structures (maps, queues, sets, lists) with automatic partitioning and replication
• Executes stream processing pipelines using a SQL-like query language or the Jet engine API
• Supports distributed computing with entry processors and submit-to-key execution patterns
• Maintains data availability through configurable synchronous or asynchronous replication
• Scales horizontally by adding nodes with automatic data rebalancing

Architecture Overview

Hazelcast uses a symmetric peer-to-peer architecture where every node in the cluster is equal. Data is partitioned across nodes using consistent hashing with 271 partitions by default, and each partition has configurable backup copies on other nodes. The embedded Jet engine handles stream processing by building a DAG of processing stages distributed across cluster members. Discovery uses multicast, TCP/IP, or cloud provider plugins (AWS, GCP, Azure, Kubernetes). Clients connect via smart routing that sends operations directly to the partition owner.

Self-Hosting & Configuration

• Run via Docker, download the distribution archive, or embed as a Maven/Gradle dependency
• Configure cluster settings in hazelcast.xml or hazelcast.yaml including network join mechanisms
• Set map configurations for time-to-live, max-idle, eviction policy, and backup count
• Enable persistence with the Hot Restart Store for crash recovery without data loss
• Deploy on Kubernetes using the official Helm chart with auto-discovery via the K8s plugin

Key Features

• Sub-millisecond read/write latency for distributed data structures
• Built-in SQL engine for querying distributed data with standard SQL syntax
• Jet stream processing engine for real-time event-driven pipelines
• Near-cache support on clients for frequently accessed data
• Client libraries for Java, Python, Node.js, Go, .NET, and C++

Comparison with Similar Tools

• Redis — simpler key-value model with richer data types; Hazelcast adds distributed computing and stream processing
• Apache Ignite — similar distributed computing platform with SQL grid, heavier resource footprint
• Apache Flink — dedicated stream processor without built-in distributed data structures
• Memcached — pure caching without replication, persistence, or compute capabilities
• Infinispan — Red Hat's distributed cache with similar Java-centric design

FAQ

Q: Is Hazelcast just a cache? A: No. While it started as a caching solution, Hazelcast now includes distributed computing, stream processing, and SQL query capabilities making it a full real-time data platform.

Q: How does Hazelcast handle node failures? A: Backup partitions are automatically promoted when a node fails. The cluster rebalances data across remaining nodes to maintain the configured backup count.

Q: Can I use Hazelcast without Java? A: Yes. Hazelcast provides thin client libraries for Python, Node.js, Go, .NET, and C++. The server itself runs on the JVM but clients can be any language.

Q: What is the difference between embedded and client-server modes? A: In embedded mode, the Hazelcast instance runs inside your application JVM and holds data. In client-server mode, dedicated Hazelcast servers hold data and your application connects as a thin client.

Sources

• https://github.com/hazelcast/hazelcast
• https://docs.hazelcast.com/