OpenObserve — Rust-Based Petabyte-Scale Observability Platform

Introduction

OpenObserve (O2) is a modern, Rust-based observability platform that targets the same job as the Loki + Tempo + Mimir stack, but ships as a single binary and stores all signals in columnar Parquet on object storage. The claim from the authors is a roughly 140x reduction in storage cost versus Elasticsearch for equivalent log volume, achieved through strong compression and minimal indexing.

What OpenObserve Does

• Accepts logs, metrics, traces, and RUM events via OTLP, Elasticsearch API, and Prometheus remote-write.
• Stores all signals in Parquet on S3, GCS, Azure Blob, or local disk.
• Provides SQL and PromQL query languages alongside a full-text search UI.
• Generates alerts with scheduled or real-time pipelines.
• Replaces Elasticsearch for many log ingestion paths via compatibility APIs.

Architecture Overview

OpenObserve runs in two modes: single-node (everything in one binary with local FS or object storage) and HA (Router, Ingester, Querier, Compactor, and AlertManager as separate processes). Ingesters write WAL then flush to Parquet with Zstandard compression; Queriers push predicates down into Parquet and scan only the columns needed. Metadata lives in etcd or sled. The query engine is DataFusion/Arrow-based, so SQL queries execute in vectorized fashion on columnar data.

Self-Hosting & Configuration

• Helm chart supports HA with MinIO or S3 backends — set ZO_S3_* env vars.
• Use ZO_META_STORE=mysql|postgres|etcd|sled to pick the metadata backend.
• ZO_DATA_DIR configures local WAL/data path; mount persistent volumes in Kubernetes.
• Enable Enterprise features (SSO, RBAC) via config flags; OSS version is feature-rich on its own.
• Vector, Fluent Bit, and OpenTelemetry Collector all ship to OpenObserve out of the box.

Key Features

• Columnar Parquet storage with ZSTD compression — dramatic cost savings for logs.
• Single Rust binary option for small deployments, microservices-scale HA for large ones.
• SQL, PromQL, and dashboard builder in one product.
• Compatible with Elasticsearch bulk API and Prometheus remote-write for drop-in migration.
• Multi-tenant, with streams scoped per organization.

Comparison with Similar Tools

• Elasticsearch / OpenSearch — higher write cost and storage footprint, great for search relevance.
• Grafana Loki + Mimir + Tempo — same three pillars, but three separate systems.
• SigNoz — similar goal, ClickHouse-based; OpenObserve stays on object storage.
• Splunk / Datadog — feature-rich SaaS at SaaS prices; OpenObserve self-hostable.
• Quickwit — Rust log engine with splits + object storage; narrower scope than O2.

FAQ

Q: Can I actually drop Elasticsearch and keep my Kibana dashboards? A: The ingest API is compatible, but Kibana itself is not — OpenObserve ships its own UI.

Q: Does it require Kafka or an external message bus? A: No. WAL + flush to object storage keeps the architecture simple.

Q: Is there a managed offering? A: Yes, the maintainers run a hosted version; everything in this post refers to the open-source build.

Q: How is tracing handled? A: OTLP traces land as Parquet rows with span-level fields; the UI renders a flame-graph view.

Sources

• https://github.com/openobserve/openobserve
• https://openobserve.ai/docs/