Quickwit — Cloud-Native Sub-Second Search Engine

Introduction

Quickwit is a distributed search engine designed for logs, traces, and other append-only data. It decouples compute from storage by indexing directly to object storage like S3, making it significantly cheaper than Elasticsearch for large-scale observability workloads.

What Quickwit Does

• Indexes structured and unstructured data to S3, MinIO, or local disk
• Provides sub-second full-text search over terabytes of log data
• Ingests data via REST API, Kafka, Kinesis, or Pulsar sources
• Supports native OpenTelemetry for logs and traces
• Offers a Jaeger-compatible gRPC endpoint for distributed tracing

Architecture Overview

Quickwit uses a stateless indexer-searcher architecture where indexers create splits (immutable chunks of indexed data) and upload them to object storage. Searchers download only the metadata catalog and fetch relevant splits on demand. A control plane coordinates cluster membership, index scheduling, and shard routing. This design lets you scale compute independently from storage.

Self-Hosting & Configuration

• Run a single binary or deploy via Docker / Helm chart on Kubernetes
• Configure data sources (Kafka, Kinesis, Pulsar, file) in YAML index configs
• Store indexes on S3, GCS, Azure Blob, or MinIO with a single storage URI
• Set retention policies to automatically delete old splits
• Enable the Jaeger gRPC endpoint for trace search integration with Grafana

Key Features

• True compute-storage separation reduces cost for large datasets
• Tantivy-based indexing engine delivers fast full-text and columnar search
• Native OpenTelemetry support for logs and traces out of the box
• Schemaless or strict schema modes for flexible data modeling
• Multi-tenant with per-index access control

Comparison with Similar Tools

• Elasticsearch — feature-rich but expensive at scale; Quickwit is 5-10x cheaper on object storage
• Grafana Loki — log aggregation focused on labels, not full-text; Quickwit offers richer search
• OpenObserve — Rust-based observability platform; Quickwit focuses on search with deeper Tantivy integration
• ClickHouse — columnar analytics DB; Quickwit is purpose-built for full-text log search
• Zinc — lightweight single-node search; Quickwit is distributed and cloud-native

FAQ

Q: Can Quickwit replace Elasticsearch? A: For log and trace search workloads, yes. It is not designed for application search with real-time updates.

Q: What object storage backends are supported? A: Amazon S3, Google Cloud Storage, Azure Blob Storage, MinIO, and any S3-compatible service.

Q: Does Quickwit support SQL queries? A: It supports a subset of SQL via its query language and is adding broader SQL support over time.

Q: How does Quickwit handle high-cardinality fields? A: It uses columnar storage for fast aggregations on high-cardinality fields within splits.

Sources

• https://github.com/quickwit-oss/quickwit
• https://quickwit.io/docs