Apache Hudi — Incremental Data Processing for Data Lakehouses

What it is

Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data lakehouse platform. It provides record-level insert, update, and delete capabilities on top of data lakes stored in S3, GCS, HDFS, or any Hadoop-compatible filesystem. Hudi turns your object storage into a mutable, transactional data store with ACID guarantees.

Hudi targets data engineers building incremental ETL pipelines, teams migrating from traditional data warehouses to lakehouse architectures, and organizations that need change data capture (CDC) ingestion from operational databases into their analytics layer.

How it saves time or tokens

Without Hudi, updating or deleting records in a data lake requires rewriting entire partitions. Hudi's record-level operations mean you only process changed records, dramatically reducing compute costs and pipeline runtime. Incremental queries let downstream consumers read only new or changed data since their last checkpoint, eliminating full-table scans.

For AI and ML workflows, Hudi's time travel feature lets you query historical snapshots of your data for reproducible training datasets without maintaining separate copies.

How to use

1. Add Hudi to your Spark, Flink, or Hive environment. For Spark: include the hudi-spark-bundle JAR in your spark-submit command.
2. Write data to a Hudi table using the Hudi datasource. Specify the record key, partition path, and precombine field.
3. Query the table using Spark SQL, Trino, or any engine that supports Hudi's metadata. Use incremental queries to process only new records.

Example

# Write to a Hudi table with upsert
df.write.format('hudi') \
  .option('hoodie.table.name', 'user_events') \
  .option('hoodie.datasource.write.recordkey.field', 'event_id') \
  .option('hoodie.datasource.write.partitionpath.field', 'event_date') \
  .option('hoodie.datasource.write.precombine.field', 'updated_at') \
  .option('hoodie.datasource.write.operation', 'upsert') \
  .mode('append') \
  .save('s3://my-lake/user_events')

# Incremental read: only changes since last checkpoint
df_incremental = spark.read.format('hudi') \
  .option('hoodie.datasource.query.type', 'incremental') \
  .option('hoodie.datasource.read.begin.instanttime', '20260401000000') \
  .load('s3://my-lake/user_events')

Related on TokRepo

• AI tools for database — Data infrastructure and database tools
• Automation tools — Pipeline orchestration and automation

Common pitfalls

• Hudi's write operations require a Spark or Flink runtime. Ensure your cluster is properly sized for the write amplification that comes with merge-on-read tables.
• Choosing between Copy-on-Write (CoW) and Merge-on-Read (MoR) table types matters. CoW is simpler and better for read-heavy workloads. MoR optimizes write performance but adds complexity to read queries.
• Metadata table and timeline management need periodic compaction and cleaning. Configure Hudi's cleaner and archiver to prevent unbounded storage growth.