# DataX — Offline Data Synchronization Tool by Alibaba

> An open-source framework for reliable, high-throughput data transfer between heterogeneous data sources. Supports MySQL, PostgreSQL, Oracle, HDFS, Hive, MongoDB, and dozens more through a pluggable reader-writer architecture.

## Install

Save in your project root:

# DataX — Offline Data Synchronization Tool by Alibaba

## Quick Use
```bash
# Download and extract DataX
tar -xzf datax.tar.gz
# Run a sync job
python datax.py /path/to/job.json
```
```json
{
  "job": {
    "content": [{
      "reader": { "name": "mysqlreader", "parameter": { "connection": [{ "jdbcUrl": ["jdbc:mysql://host:3306/db"], "table": ["users"] }], "username": "root", "password": "***", "column": ["*"] } },
      "writer": { "name": "hdfswriter", "parameter": { "path": "/data/users", "fileType": "orc", "column": [{ "name": "id", "type": "bigint" }] } }
    }],
    "setting": { "speed": { "channel": 4 } }
  }
}
```

## Introduction
DataX is Alibaba's open-source offline data synchronization framework used internally to move petabytes of data daily between relational databases, data warehouses, HDFS, and other storage systems. It uses a pluggable reader-writer architecture where each data source is an independent plugin.

## What DataX Does
- Transfers data between any combination of supported sources and destinations
- Scales throughput by running multiple parallel channels within a single job
- Supports 20+ data source plugins including MySQL, PostgreSQL, Oracle, SQL Server, HDFS, Hive, HBase, MongoDB, and Elasticsearch
- Provides dirty-data handling with configurable thresholds for skipping bad records
- Runs as a standalone process with no external dependencies beyond Java and Python

## Architecture Overview
DataX follows a Framework + Plugin design. The core framework manages job lifecycle, scheduling, memory buffering, and flow control. Each data source implements a Reader plugin (to extract) and a Writer plugin (to load). Data flows through an in-memory channel that decouples readers from writers, allowing different source and destination speeds. A transformer layer can apply column-level transformations mid-flight. The job is configured as a single JSON file describing the reader, writer, and channel settings.

## Self-Hosting & Configuration
- Requires Java 8+ and Python 2.7+; extract the tarball and run — no installation needed
- Define sync jobs as JSON files specifying reader, writer, and speed settings
- Tune throughput by adjusting the channel count and memory-per-channel parameters
- Configure dirty-data limits to skip or fail on records that cannot be converted
- Add custom plugins by implementing the Reader or Writer interface and dropping JARs in the plugin directory

## Key Features
- Moves petabytes daily inside Alibaba, proven at massive scale
- Plugin architecture means adding a new data source requires only a reader or writer JAR
- In-memory channel with back-pressure keeps both sides running at optimal speed
- Built-in statistics reporting shows records transferred, bytes moved, and error counts
- Standalone execution with no external orchestrator required for simple jobs

## Comparison with Similar Tools
- **Apache NiFi** — visual dataflow platform for real-time and batch; DataX is simpler and batch-focused with JSON job configs
- **Apache SeaTunnel** — next-generation data integration engine; offers a richer connector ecosystem and distributed execution
- **Airbyte** — cloud-native EL(T) platform with a UI and connector marketplace; DataX is lighter-weight and runs as a CLI tool
- **dbt** — transforms data already in the warehouse; DataX moves data between systems rather than transforming in place
- **Debezium** — change data capture for real-time streaming; DataX handles batch/offline synchronization

## FAQ
**Q: Can DataX handle real-time streaming?**
A: No. DataX is designed for batch/offline data synchronization. For real-time CDC, consider Debezium or Apache Flink.

**Q: How many data sources does DataX support?**
A: Over 20 official plugins covering relational databases, NoSQL stores, HDFS, object storage, and search engines.

**Q: Does DataX require a cluster to run?**
A: No. It runs as a single-node process. For distributed scheduling, pair it with an orchestrator like DolphinScheduler or Airflow.

**Q: Is DataX still maintained?**
A: Yes. The repository is actively maintained with community contributions and periodic releases.

## Sources
- https://github.com/alibaba/DataX
- https://github.com/alibaba/DataX/blob/master/introduction.md

---
Source: https://tokrepo.com/en/workflows/asset-5588c295
Author: AI Open Source