Kedro — Production-Ready ML Pipeline Framework for Python

What it is

Kedro is an open-source Python framework created by McKinsey QuantumBlack that applies software engineering best practices to data science and machine learning code. It provides a standardized project structure, a declarative data catalog, and a pipeline abstraction that transforms experimental notebook code into maintainable, testable, production-ready pipelines.

Kedro targets data scientists and ML engineers who need to bridge the gap between prototype notebooks and production systems. It works alongside existing tools like pandas, scikit-learn, and PySpark without replacing them.

How it saves time or tokens

Kedro eliminates the 'notebook to production' refactoring cycle. The standardized project template means new team members understand the codebase layout immediately. The data catalog decouples data access from business logic, so switching between local CSV files and cloud storage requires changing a YAML config, not Python code. Pipeline visualization with kedro viz provides instant documentation of data flow without writing diagram code.

How to use

1. Install Kedro and create a new project:

pip install kedro
kedro new --starter=spaceflights-pandas
cd spaceflights-pandas

1. Run the pipeline:

kedro run

1. Visualize the pipeline graph:

kedro viz run
# Opens browser at localhost:4141

Example

Define a pipeline node that transforms data:

# src/project/pipelines/data_processing/nodes.py
import pandas as pd

def preprocess_companies(companies: pd.DataFrame) -> pd.DataFrame:
    companies['company_rating'] = companies['company_rating'].fillna(
        companies['company_rating'].mean()
    )
    return companies

Register it in the pipeline:

# src/project/pipelines/data_processing/pipeline.py
from kedro.pipeline import Pipeline, node
from .nodes import preprocess_companies

def create_pipeline(**kwargs) -> Pipeline:
    return Pipeline([
        node(
            func=preprocess_companies,
            inputs='companies',
            outputs='preprocessed_companies',
            name='preprocess_companies_node',
        ),
    ])

The data catalog YAML maps logical names to physical storage.

Related on TokRepo

• AI Tools for Coding — Development tools that complement ML pipeline frameworks
• AI Tools for DevOps — CI/CD and deployment tools for ML pipeline orchestration

Common pitfalls

• Kedro is a pipeline framework, not an orchestrator. For scheduled execution, pair it with Airflow, Prefect, or Argo using Kedro's deployment plugins.
• The data catalog requires explicit registration of every dataset. Forgetting to add an intermediate dataset to catalog.yml causes runtime errors.
• Pipeline visualization with kedro viz requires installing the kedro-viz plugin separately. It is not included in the core package.
• Always check the official documentation for the latest version-specific changes and migration guides before upgrading in production environments.
• For team deployments, establish clear guidelines on configuration and usage patterns to ensure consistency across developers.