# Azkaban — Batch Workflow Scheduler by LinkedIn

> A Hadoop-era workflow scheduler with a web UI for defining, scheduling, and monitoring interdependent batch jobs with dependency resolution.

## Install

Save in your project root:

# Azkaban — Batch Workflow Scheduler by LinkedIn

## Quick Use
```bash
git clone https://github.com/azkaban/azkaban.git
cd azkaban && ./gradlew build -x test
# Start solo server (all-in-one mode)
cd azkaban-solo-server/build/install/azkaban-solo-server
bin/start-solo.sh
# Web UI at http://localhost:8081
```

## Introduction
Azkaban is a batch workflow job scheduler created at LinkedIn for managing Hadoop jobs. It provides a web-based interface for uploading workflow definitions, scheduling them on cron-like triggers, and monitoring execution with dependency-aware ordering. While originally built for Hadoop, it can orchestrate any command-line job.

## What Azkaban Does
- Defines workflows as directed acyclic graphs (DAGs) of interdependent jobs
- Schedules workflows via cron expressions or one-time triggers through the web UI
- Resolves job dependencies automatically so each job runs only after its prerequisites succeed
- Provides real-time log streaming and execution history through the web interface
- Supports SLA alerts that notify when jobs exceed expected duration

## Architecture Overview
Azkaban consists of three components: a relational database (MySQL) for state, a web server for the UI and REST API, and one or more executor servers that run jobs. Workflows are defined as `.job` property files packaged in ZIP archives. The web server handles scheduling and dispatches flows to executors. Executors run jobs as OS processes and report status back through the database. A multi-executor mode distributes work across a pool of executor servers.

## Self-Hosting & Configuration
- Requires Java 8+ and MySQL for the metadata store
- Build from source with Gradle or use the solo-server mode for evaluation
- Configure database connection in `azkaban.properties`
- Upload workflow ZIP packages through the web UI or REST API
- Set up SLA rules and email notifications per project or flow

## Key Features
- Web UI with project management, execution graphs, and log viewing
- Job type plugins for Hadoop, Pig, Hive, Spark, and generic command execution
- Flow-level and job-level retries with configurable backoff
- Permission model with project-level access control for teams
- REST API for programmatic workflow management and monitoring

## Comparison with Similar Tools
- **Apache Airflow** — Python-based DAG scheduler with a larger ecosystem; Azkaban uses property files and ZIP uploads, simpler for ops teams
- **Luigi** — Python library for pipeline definition; Azkaban provides a standalone server with web UI and scheduling
- **Oozie** — XML-based Hadoop workflow engine; Azkaban offers a more user-friendly interface and simpler job definitions
- **Dagster** — Modern data orchestration with asset-centric approach; Azkaban is a traditional job scheduler without data lineage features

## FAQ
**Q: Is Azkaban still actively maintained?**
A: Development has slowed, but the project remains functional and is used in production at several organizations. Consider Airflow for greenfield projects.

**Q: Can Azkaban run non-Hadoop jobs?**
A: Yes, the command job type can execute any shell command, making it usable for general-purpose workflow scheduling.

**Q: How do I scale Azkaban?**
A: Deploy multiple executor servers and configure the web server to distribute flows across them using the multi-executor mode.

**Q: Does Azkaban support dynamic workflows?**
A: No, workflows are statically defined in ZIP packages. For dynamic DAG generation, consider Apache Airflow.

## Sources
- https://github.com/azkaban/azkaban
- https://azkaban.readthedocs.io/

---
Source: https://tokrepo.com/en/workflows/asset-1d95b8bf
Author: AI Open Source