Dagster — Cloud-Native Data Pipeline Orchestrator

What it is

Dagster is an open-source data pipeline orchestrator built around the concept of software-defined assets. Instead of defining tasks and their execution order, you define the data assets your pipeline produces and Dagster figures out the dependency graph, execution plan, and lineage automatically. It is written in Python and supports both local development and cloud deployment.

Dagster targets data engineers who want testable, observable pipelines with clear data lineage. It competes with Airflow, Prefect, and Mage as a modern orchestration layer that treats data assets as first-class citizens.

How it saves time or tokens

Dagster's asset-based approach eliminates the gap between what your pipeline produces and how it runs. Each asset is a Python function with typed inputs and outputs, which means you can unit test individual assets locally before deploying. The built-in asset catalog shows what data exists, when it was last materialized, and what downstream assets depend on it.

For AI/ML teams, Dagster provides native integration with ML frameworks, making it straightforward to orchestrate training data preparation, model training, and evaluation as linked assets.

How to use

1. Install Dagster: pip install dagster dagster-webserver. Create a new project with dagster project scaffold --name my_pipeline.
2. Define assets as Python functions decorated with @asset. Each function takes upstream assets as inputs and returns the produced data.
3. Launch the Dagster UI: dagster dev. View your asset graph, materialize assets on-demand, and schedule recurring materializations.

Example

from dagster import asset, Definitions
import pandas as pd

@asset
def raw_orders() -> pd.DataFrame:
    return pd.read_csv('https://example.com/orders.csv')

@asset
def daily_summary(raw_orders: pd.DataFrame) -> pd.DataFrame:
    return raw_orders.groupby('date').agg(
        total_revenue=('amount', 'sum'),
        order_count=('order_id', 'count')
    ).reset_index()

defs = Definitions(assets=[raw_orders, daily_summary])

Dagster infers that daily_summary depends on raw_orders from the function signature. The asset graph is visualized in the web UI.

Related on TokRepo

• AI tools for automation — Pipeline and workflow automation tools
• AI tools for database — Data infrastructure and storage tools

Common pitfalls

• Migrating from Airflow requires rethinking your pipeline structure. Airflow uses task DAGs while Dagster uses asset definitions. The mental model shift takes time but results in cleaner pipelines.
• The asset graph grows complex in large organizations. Use asset groups and code locations to organize assets by team or domain.
• Dagster Cloud is the managed deployment option. Self-hosting requires running the daemon, webserver, and a database (PostgreSQL recommended) for production workloads.