dlt — Data Load Tool for Python ELT Pipelines

What it is

dlt (data load tool) is an open-source Python library that simplifies building ELT pipelines. You define a data source as a Python generator or function, choose a destination (BigQuery, Snowflake, DuckDB, PostgreSQL, and others), and dlt handles schema inference, incremental loading, data normalization, and state management automatically.

dlt targets data engineers and analysts who want to build production data pipelines in pure Python without learning a new framework or YAML DSL. It is lightweight, embeddable, and works in scripts, notebooks, and orchestrators alike.

How it saves time or tokens

dlt eliminates the boilerplate of data loading: schema creation, type mapping, incremental state tracking, and nested JSON flattening. A pipeline that would take hundreds of lines with raw SQL and API calls becomes a few lines of Python. The automatic schema inference means you do not need to pre-define table schemas; dlt creates and evolves them based on the data it sees.

For AI workflows, dlt makes it easy to load API responses from LLM providers, vector databases, or analytics services into a data warehouse for analysis and reporting.

How to use

1. Install dlt with your destination: pip install dlt[bigquery] (or dlt[duckdb], dlt[snowflake], etc.).
2. Define a source function that yields data. Use @dlt.resource to mark it as a loadable resource.
3. Create a pipeline, connect it to your destination, and run it. dlt infers the schema, creates tables, and loads the data.

Example

import dlt
import requests

@dlt.resource(write_disposition='merge', primary_key='id')
def github_issues():
    response = requests.get(
        'https://api.github.com/repos/dlt-hub/dlt/issues',
        params={'state': 'open', 'per_page': 100}
    )
    yield response.json()

pipeline = dlt.pipeline(
    pipeline_name='github_pipeline',
    destination='duckdb',
    dataset_name='github_data'
)

load_info = pipeline.run(github_issues)
print(load_info)

This pipeline fetches GitHub issues, creates a DuckDB table with inferred schema, and merges new data on subsequent runs using the id primary key.

Related on TokRepo

• AI tools for database — Data infrastructure and database tools
• Automation tools — Pipeline orchestration and automation

Common pitfalls

• Schema inference works well for consistent data shapes. Highly variable JSON structures may produce wide tables with many nullable columns. Use dlt.resource hints to control column selection.
• Incremental loading requires a cursor field (like updated_at). Without it, dlt loads all data on each run. Set incremental=dlt.sources.incremental('updated_at') to enable incremental behavior.
• dlt pipelines run in-process by default. For large-scale production workloads, run them inside an orchestrator (Dagster, Airflow, Prefect) for scheduling, monitoring, and retry handling.