Apache Pinot — Real-Time Distributed OLAP Datastore

Introduction

Apache Pinot is built specifically for OLAP queries that need to return in milliseconds even over billions of rows. Unlike traditional data warehouses that optimize for batch analytics, Pinot is designed for user-facing applications where thousands of concurrent queries must complete fast. It ingests data in real-time from streaming sources like Apache Kafka alongside batch loads from data lakes.

What Apache Pinot Does

• Ingests data in real-time from Kafka, Kinesis, and Pulsar with sub-second availability
• Serves analytical SQL queries with millisecond latency over billions of records
• Supports star-tree indexing for pre-aggregated fast lookups on common query patterns
• Provides a multi-tenant architecture where tables are independently configured and scaled
• Handles both real-time and offline (batch) data segments with automatic merge and retention

Architecture Overview

Pinot has four main components: Controllers manage cluster metadata, Brokers route queries, Servers store and process data segments, and Minions handle background tasks like segment merge and purge. Data is divided into segments that are distributed across servers. Real-time segments consume from streaming sources and are periodically converted to immutable offline segments. Queries are scattered to relevant servers and gathered at the broker for a merged response.

Self-Hosting & Configuration

• Deploy via Docker, Kubernetes Helm charts, or compile from source with Java 11+
• Requires Apache ZooKeeper for cluster coordination and metadata storage
• Configure tables with a JSON schema defining columns, indexes, and ingestion sources
• Set up real-time tables with a Kafka consumer config for streaming ingestion
• Tune segment size, retention, and replication factor per table for performance and durability

Key Features

• Star-tree index pre-computes aggregations for common group-by queries with constant-time lookups
• Inverted index, range index, bloom filter, and text index for flexible query optimization
• Pluggable stream ingestion supporting Kafka, Kinesis, and custom connectors
• Tiered storage moves older segments to cost-effective storage while keeping hot data on SSD
• Multi-stage query engine supports joins and complex SQL across distributed tables

Comparison with Similar Tools

• Apache Druid — Similar real-time OLAP design; Pinot has tighter Kafka integration and star-tree indexes
• ClickHouse — Column-store with broader SQL support; stronger for batch analytics, Pinot optimized for concurrent real-time queries
• Apache Doris — MySQL-compatible OLAP; simpler setup but fewer indexing strategies than Pinot
• StarRocks — Fork of Doris with performance focus; Pinot has more mature streaming ingestion
• Elasticsearch — Full-text search with aggregations; Pinot is faster for structured OLAP queries at scale

FAQ

Q: What query language does Pinot use? A: Pinot supports SQL through its multi-stage query engine, including SELECT, GROUP BY, ORDER BY, joins, and window functions.

Q: Can Pinot replace my data warehouse? A: Pinot is optimized for low-latency queries on pre-defined schemas. For ad-hoc exploration and complex ETL, pair it with a data warehouse.

Q: How does it handle schema changes? A: Pinot supports adding new columns to existing tables. Existing segments can be backfilled with default values or reprocessed.

Q: What scale does Pinot support? A: Pinot clusters in production handle trillions of events, petabytes of data, and hundreds of thousands of queries per second.

Sources

• https://github.com/apache/pinot
• https://docs.pinot.apache.org/