ZenML — MLOps Pipeline Framework from Development to Production

Introduction

ZenML is an MLOps framework that decouples ML pipeline logic from infrastructure. You write pipelines as Python functions and ZenML handles execution on local machines, Kubernetes, or cloud services. It integrates with existing tools rather than replacing them.

What ZenML Does

• Defines ML pipelines as decorated Python functions with typed inputs and outputs
• Runs the same pipeline code locally, on Kubernetes, or on cloud orchestrators
• Tracks artifacts, models, and metadata across pipeline runs automatically
• Integrates with 30+ MLOps tools (MLflow, W&B, Seldon, BentoML, etc.) via a stack system
• Provides a web dashboard for monitoring pipelines, artifacts, and model versions

Architecture Overview

ZenML uses a stack abstraction where each component (orchestrator, artifact store, experiment tracker, model deployer) is pluggable. The ZenML server stores pipeline metadata in a database. The client serializes pipeline code and ships it to the configured orchestrator for execution.

Self-Hosting & Configuration

• Install via pip; optionally deploy the ZenML server via Docker or Helm
• Configure stacks that map to your infrastructure (e.g., Airflow + S3 + MLflow)
• Use the CLI or dashboard to switch between local and cloud stacks
• Supports secrets management for credentials across stack components
• Deploy the server on Kubernetes for multi-user team environments

Key Features

• Infrastructure-agnostic: same pipeline runs locally or on Kubernetes, Vertex AI, Sagemaker, or AzureML
• Pluggable stack components avoid lock-in to any single MLOps vendor
• Built-in model registry tracks model versions and deployment status
• Artifact versioning and lineage without additional setup
• Template system for bootstrapping common ML project patterns

Comparison with Similar Tools

• MLflow — experiment tracking and model registry; ZenML provides the pipeline orchestration layer on top
• Kedro — pipeline framework; ZenML adds production deployment and infrastructure abstraction
• Metaflow — Netflix's workflow manager; ZenML offers broader orchestrator support and a pluggable stack model
• Flyte — Kubernetes-native orchestration; ZenML runs on more orchestrators and integrates more MLOps tools
• Kubeflow — ML platform for K8s; ZenML is lighter weight and not tied to Kubernetes

FAQ

Q: Do I need Kubernetes? A: No. ZenML runs locally, on Docker, or on cloud orchestrators. Kubernetes is just one option.

Q: How does the stack system work? A: A stack is a combination of components (orchestrator, artifact store, etc.) that defines where and how pipelines run. Switch stacks to change infrastructure.

Q: Can I use ZenML with existing MLflow experiments? A: Yes. ZenML has an MLflow integration that logs experiments to your MLflow server as part of pipeline runs.

Q: Is there a managed offering? A: Yes. ZenML Cloud provides a hosted server with team management and enhanced features.

Sources

• https://github.com/zenml-io/zenml
• https://docs.zenml.io/