Azkaban — Batch Workflow Scheduler by LinkedIn

Introduction

Azkaban is a batch workflow job scheduler created at LinkedIn for managing Hadoop jobs. It provides a web-based interface for uploading workflow definitions, scheduling them on cron-like triggers, and monitoring execution with dependency-aware ordering. While originally built for Hadoop, it can orchestrate any command-line job.

What Azkaban Does

• Defines workflows as directed acyclic graphs (DAGs) of interdependent jobs
• Schedules workflows via cron expressions or one-time triggers through the web UI
• Resolves job dependencies automatically so each job runs only after its prerequisites succeed
• Provides real-time log streaming and execution history through the web interface
• Supports SLA alerts that notify when jobs exceed expected duration

Architecture Overview

Azkaban consists of three components: a relational database (MySQL) for state, a web server for the UI and REST API, and one or more executor servers that run jobs. Workflows are defined as .job property files packaged in ZIP archives. The web server handles scheduling and dispatches flows to executors. Executors run jobs as OS processes and report status back through the database. A multi-executor mode distributes work across a pool of executor servers.

Self-Hosting & Configuration

• Requires Java 8+ and MySQL for the metadata store
• Build from source with Gradle or use the solo-server mode for evaluation
• Configure database connection in azkaban.properties
• Upload workflow ZIP packages through the web UI or REST API
• Set up SLA rules and email notifications per project or flow

Key Features

• Web UI with project management, execution graphs, and log viewing
• Job type plugins for Hadoop, Pig, Hive, Spark, and generic command execution
• Flow-level and job-level retries with configurable backoff
• Permission model with project-level access control for teams
• REST API for programmatic workflow management and monitoring

Comparison with Similar Tools

• Apache Airflow — Python-based DAG scheduler with a larger ecosystem; Azkaban uses property files and ZIP uploads, simpler for ops teams
• Luigi — Python library for pipeline definition; Azkaban provides a standalone server with web UI and scheduling
• Oozie — XML-based Hadoop workflow engine; Azkaban offers a more user-friendly interface and simpler job definitions
• Dagster — Modern data orchestration with asset-centric approach; Azkaban is a traditional job scheduler without data lineage features

FAQ

Q: Is Azkaban still actively maintained? A: Development has slowed, but the project remains functional and is used in production at several organizations. Consider Airflow for greenfield projects.

Q: Can Azkaban run non-Hadoop jobs? A: Yes, the command job type can execute any shell command, making it usable for general-purpose workflow scheduling.

Q: How do I scale Azkaban? A: Deploy multiple executor servers and configure the web server to distribute flows across them using the multi-executor mode.

Q: Does Azkaban support dynamic workflows? A: No, workflows are statically defined in ZIP packages. For dynamic DAG generation, consider Apache Airflow.

Sources

• https://github.com/azkaban/azkaban
• https://azkaban.readthedocs.io/