Delta Lake — Reliable Data Lakehouse Storage Layer

Introduction

Delta Lake adds reliability to data lakes by layering ACID transactions and schema enforcement on top of Parquet files stored in cloud object storage. It lets data engineers treat a data lake like a database with full transactional guarantees.

What Delta Lake Does

• Provides ACID transactions for concurrent reads and writes on data lakes
• Maintains a transaction log that enables time travel to any previous version
• Enforces and evolves schema automatically on write operations
• Supports MERGE, UPDATE, and DELETE operations on large-scale datasets
• Integrates with Spark, Flink, Trino, Presto, and standalone Rust/Python readers

Architecture Overview

Delta Lake stores data as Parquet files alongside a JSON-based transaction log in the _delta_log directory. Each commit appends a new log entry describing file additions and removals. Readers reconstruct table state by replaying the log, with periodic checkpoints accelerating this process. Optimistic concurrency control handles conflicting writes without locking.

Self-Hosting & Configuration

• Add the delta-spark package to your Spark cluster or use delta-rs for standalone access
• Tables live on S3, GCS, Azure Blob, or HDFS with no special server required
• Enable auto-compaction with delta.autoOptimize.optimizeWrite = true
• Configure retention periods for VACUUM to clean up old Parquet files
• Use delta.logRetentionDuration to control how far back time travel works

Key Features

• Time travel lets you query or restore any historical version of a table
• Z-ordering and data skipping accelerate queries on high-cardinality columns
• Change Data Feed captures row-level changes for downstream pipelines
• Liquid clustering replaces static partitioning with adaptive data layout
• UniForm allows Delta tables to be read as Apache Iceberg or Hudi tables

Comparison with Similar Tools

• Apache Iceberg — similar table format with broader engine support; Delta has deeper Spark integration
• Apache Hudi — focuses on incremental processing and upserts; Delta is more batch-oriented
• Apache Parquet — a file format only; Delta adds transactions and metadata on top of Parquet
• Apache ORC — alternative columnar format; Delta uses Parquet as its underlying storage
• LakeFS — provides Git-like versioning for data lakes; Delta handles versioning via its transaction log

FAQ

Q: Do I need Databricks to use Delta Lake? A: No. Delta Lake is fully open source and works with any Spark deployment, standalone Python via delta-rs, or engines like Flink and Trino.

Q: How does time travel work? A: The transaction log records every change. You query a past version by specifying a version number or timestamp, and Delta replays the log to that point.

Q: Can Delta Lake handle streaming data? A: Yes. Structured Streaming in Spark can write to and read from Delta tables as both a sink and source.

Q: What is the delta-rs project? A: A native Rust implementation of the Delta Lake protocol that enables reading and writing Delta tables without Spark, with bindings for Python and Node.js.

Sources

• https://github.com/delta-io/delta
• https://docs.delta.io/latest/index.html