Polyaxon — ML Lifecycle Management and Orchestration Platform

Introduction

Polyaxon is an open-source MLOps platform that helps teams manage experiments, automate hyperparameter tuning, and orchestrate ML pipelines on Kubernetes. It provides a unified interface for tracking runs, comparing results, and deploying models across the full machine learning lifecycle.

What Polyaxon Does

• Tracks experiments with automatic logging of metrics, parameters, and artifacts
• Runs distributed hyperparameter searches using grid, random, Bayesian, and Hyperband methods
• Orchestrates multi-step ML pipelines as DAGs on Kubernetes
• Manages compute resources with scheduling and quota policies
• Provides a web dashboard for comparing experiments and visualizing results

Architecture Overview

Polyaxon runs on Kubernetes as a set of microservices. The API server handles experiment submissions and metadata storage. A scheduler allocates jobs to cluster resources based on queue priority and resource requests. The sidecar agent monitors running experiments and streams logs. Artifacts are stored in configured object storage (S3, GCS, Azure Blob), while metadata lives in PostgreSQL.

Self-Hosting & Configuration

• Deploy on Kubernetes via Helm: helm install polyaxon polyaxon/polyaxon
• Requires PostgreSQL, RabbitMQ (or Redis), and object storage for artifacts
• Configure access with polyaxon config set --host=https://polyaxon.example.com
• Define experiments in YAML polyaxonfiles specifying environment, code, and hyperparameters
• Supports GPU scheduling with NVIDIA device plugin integration

Key Features

• Native hyperparameter optimization with early stopping via Hyperband and median stopping
• DAG-based pipeline orchestration for multi-step ML workflows
• Built-in Jupyter notebook and TensorBoard spawning from the dashboard
• Multi-tenant with role-based access control and project isolation
• Supports PyTorch, TensorFlow, MXNet, and any containerized workload

Comparison with Similar Tools

• MLflow — lightweight experiment tracking; Polyaxon adds Kubernetes-native orchestration and scheduling
• Kubeflow — broader Kubernetes ML platform; Polyaxon offers a more opinionated and integrated experience
• Weights & Biases — SaaS experiment tracking; Polyaxon is fully self-hosted with pipeline orchestration
• Determined AI — focused on training; Polyaxon covers the full lifecycle from experimentation to deployment

FAQ

Q: Can I use Polyaxon without Kubernetes? A: The open-source version requires Kubernetes. Polyaxon CE provides a Docker Compose option for local testing.

Q: How does Polyaxon handle distributed training? A: It supports native distributed training for PyTorch (DDP), TensorFlow, MPI, and Horovod via Kubernetes job scheduling.

Q: Is there a hosted cloud version? A: Yes. Polyaxon Cloud offers a managed service, but the open-source version can be fully self-hosted.

Q: How do I migrate from MLflow to Polyaxon? A: Polyaxon can run MLflow tracking as a component. Migrate experiments gradually by pointing new runs to the Polyaxon tracking server.

Sources

• https://github.com/polyaxon/polyaxon
• https://polyaxon.com/docs