DataHub — Open-Source Data Discovery & Governance Platform

What it is

DataHub is a modern metadata platform originally built inside LinkedIn and now maintained by Acryl Data. It provides a single pane of glass for discovering, governing, and observing data assets across your entire stack. The platform ingests metadata automatically from 50+ sources including Snowflake, BigQuery, dbt, Airflow, Spark, and Kafka.

DataHub is built for data engineers, analysts, and platform teams who need to understand what data exists, where it flows, and who owns it. It replaces tribal knowledge with a searchable catalog that tracks lineage, enforces policies, and surfaces data quality signals.

How it saves time or tokens

DataHub eliminates the 'ask around' pattern for finding data. Instead of messaging five people to locate a table or understand a column's meaning, you search the catalog. Column-level lineage tells you exactly which upstream jobs feed a dashboard metric, cutting incident investigation from hours to minutes. Automated ingestion recipes run on schedule, keeping the catalog current without manual effort.

How to use

1. Install the CLI and start DataHub locally:

python3 -m pip install --upgrade acryl-datahub
datahub docker quickstart

1. Access the UI at http://localhost:9002 with default credentials (datahub/datahub).

1. Create an ingestion recipe YAML for your data source and run it:

datahub ingest -c my_recipe.yml

Example

A basic ingestion recipe for a PostgreSQL database:

source:
  type: postgres
  config:
    host_port: 'localhost:5432'
    database: 'analytics'
    username: 'datahub'
    password: '${POSTGRES_PASSWORD}'

sink:
  type: datahub-rest
  config:
    server: 'http://localhost:8080'

This recipe connects to a Postgres instance, extracts table schemas, column descriptions, and relationships, then pushes the metadata to your DataHub instance.

Related on TokRepo

• AI Tools for Database — Database management and query tools that pair well with DataHub
• AI Tools for Documentation — Documentation generators for data catalogs and API references

Common pitfalls

• The Docker quickstart requires Docker Compose and allocates significant memory. Ensure at least 8GB available RAM before starting.
• Ingestion recipes need explicit schema filtering for large warehouses. Without filters, initial ingestion can take hours on databases with thousands of tables.
• DataHub uses Kafka internally for its stream-first architecture. In production deployments, monitoring Kafka lag is critical for metadata freshness.
• Always check the official documentation for the latest version-specific changes and migration guides before upgrading in production environments.
• For team deployments, establish clear guidelines on configuration and usage patterns to ensure consistency across developers.