DataX — Offline Data Synchronization Tool by Alibaba

Introduction

DataX is Alibaba's open-source offline data synchronization framework used internally to move petabytes of data daily between relational databases, data warehouses, HDFS, and other storage systems. It uses a pluggable reader-writer architecture where each data source is an independent plugin.

What DataX Does

• Transfers data between any combination of supported sources and destinations
• Scales throughput by running multiple parallel channels within a single job
• Supports 20+ data source plugins including MySQL, PostgreSQL, Oracle, SQL Server, HDFS, Hive, HBase, MongoDB, and Elasticsearch
• Provides dirty-data handling with configurable thresholds for skipping bad records
• Runs as a standalone process with no external dependencies beyond Java and Python

Architecture Overview

DataX follows a Framework + Plugin design. The core framework manages job lifecycle, scheduling, memory buffering, and flow control. Each data source implements a Reader plugin (to extract) and a Writer plugin (to load). Data flows through an in-memory channel that decouples readers from writers, allowing different source and destination speeds. A transformer layer can apply column-level transformations mid-flight. The job is configured as a single JSON file describing the reader, writer, and channel settings.

Self-Hosting & Configuration

• Requires Java 8+ and Python 2.7+; extract the tarball and run — no installation needed
• Define sync jobs as JSON files specifying reader, writer, and speed settings
• Tune throughput by adjusting the channel count and memory-per-channel parameters
• Configure dirty-data limits to skip or fail on records that cannot be converted
• Add custom plugins by implementing the Reader or Writer interface and dropping JARs in the plugin directory

Key Features

• Moves petabytes daily inside Alibaba, proven at massive scale
• Plugin architecture means adding a new data source requires only a reader or writer JAR
• In-memory channel with back-pressure keeps both sides running at optimal speed
• Built-in statistics reporting shows records transferred, bytes moved, and error counts
• Standalone execution with no external orchestrator required for simple jobs

Comparison with Similar Tools

• Apache NiFi — visual dataflow platform for real-time and batch; DataX is simpler and batch-focused with JSON job configs
• Apache SeaTunnel — next-generation data integration engine; offers a richer connector ecosystem and distributed execution
• Airbyte — cloud-native EL(T) platform with a UI and connector marketplace; DataX is lighter-weight and runs as a CLI tool
• dbt — transforms data already in the warehouse; DataX moves data between systems rather than transforming in place
• Debezium — change data capture for real-time streaming; DataX handles batch/offline synchronization

FAQ

Q: Can DataX handle real-time streaming? A: No. DataX is designed for batch/offline data synchronization. For real-time CDC, consider Debezium or Apache Flink.

Q: How many data sources does DataX support? A: Over 20 official plugins covering relational databases, NoSQL stores, HDFS, object storage, and search engines.

Q: Does DataX require a cluster to run? A: No. It runs as a single-node process. For distributed scheduling, pair it with an orchestrator like DolphinScheduler or Airflow.

Q: Is DataX still maintained? A: Yes. The repository is actively maintained with community contributions and periodic releases.

Sources

• https://github.com/alibaba/DataX
• https://github.com/alibaba/DataX/blob/master/introduction.md