Conductor — Scalable Workflow Orchestration Engine

Introduction

Conductor is a workflow orchestration engine that coordinates microservices and long-running processes. Originally developed at Netflix to handle millions of workflow executions daily, it provides durable execution, retry logic, and a visual UI for monitoring distributed workflows across services.

What Conductor Does

• Orchestrates complex microservice workflows defined as JSON or code
• Provides durable execution with automatic retries and error handling
• Offers a web-based UI for workflow visualization and debugging
• Supports human-in-the-loop tasks, sub-workflows, and dynamic forking
• Integrates with external systems via HTTP, gRPC, and event-driven workers

Architecture Overview

Conductor uses a server-worker model. The server maintains workflow state in a persistent store (Redis, PostgreSQL, or MySQL) and an indexing backend (Elasticsearch or OpenSearch). Workers poll task queues, execute business logic, and report results. The server advances the workflow DAG based on task outcomes, handling retries, timeouts, and conditional branching. A REST API and UI provide full control over workflow definitions and executions.

Self-Hosting & Configuration

• Deploy via Docker Compose or Kubernetes Helm charts from the official repo
• Configure persistence with PostgreSQL, MySQL, or Redis for production workloads
• Set up Elasticsearch or OpenSearch for workflow search and indexing
• Define workflows in JSON or use the Java, Python, Go, or C# SDKs
• Secure the API with OAuth2 or custom authentication middleware

Key Features

• Handles millions of concurrent workflow executions with horizontal scaling
• Visual workflow designer and execution timeline in the built-in UI
• Supports SIMPLE, HTTP, FORK_JOIN, DECISION, SUB_WORKFLOW, and custom task types
• Provides SDKs for Java, Python, Go, C#, JavaScript, and Clojure
• Event-driven architecture with SQS, NATS, and AMQP integration

Comparison with Similar Tools

• Temporal — similar durable execution but uses code-first SDK approach instead of JSON definitions
• Apache Airflow — DAG-based scheduler focused on data pipelines rather than microservice orchestration
• Prefect — Python-native workflow engine with less emphasis on multi-language workers
• Step Functions — AWS-managed state machine, vendor-locked but zero infrastructure
• Cadence — Uber's predecessor to Temporal with overlapping feature set

FAQ

Q: How does Conductor differ from Temporal? A: Conductor uses JSON workflow definitions and a task-queue polling model, while Temporal uses code-first workflows with a replay-based execution model. Conductor provides a built-in UI out of the box.

Q: Can Conductor handle long-running workflows? A: Yes. Workflows can run for days or months. Conductor persists state and resumes execution after failures or restarts.

Q: What databases does Conductor support? A: PostgreSQL, MySQL, and Redis are supported as persistence backends. Elasticsearch or OpenSearch handle workflow indexing and search.

Q: Is Conductor suitable for data pipelines? A: It can orchestrate data pipelines, but tools like Airflow or Dagster may be better suited for DAG-based ETL scheduling with built-in data-aware features.

Sources

• https://github.com/conductor-oss/conductor
• https://docs.conductor-oss.org/