# Kestra — Event-Driven Orchestration & Scheduling Platform

> Kestra is an open-source orchestration platform for scheduling and running complex data pipelines, ETL jobs, and automation workflows with declarative YAML.

## Install

Save the content below to `.claude/skills/` or append to your `CLAUDE.md`:

## Quick Use

```bash
docker run --pull=always --rm -it -p 8080:8080 -p 8081:8081 kestra/kestra:latest server local
```

Open `http://localhost:8080` — create your first flow in the built-in editor.

## Intro

**Kestra** is an open-source event-driven orchestration and scheduling platform designed for mission-critical data pipelines and automation workflows. Built with Java and featuring a declarative YAML-based approach, it makes complex workflow orchestration accessible to both engineers and non-technical users.

With 26.7K+ GitHub stars and Apache-2.0 license, Kestra combines the power of code-based orchestration with a visual low-code interface, supporting 500+ plugins for integrations with databases, cloud services, messaging systems, and more.

## What Kestra Does

Kestra handles workflow orchestration across multiple domains:

- **Data Pipelines**: Schedule and run ETL/ELT jobs with built-in support for dbt, Spark, Flink, and major databases
- **Event-Driven Automation**: React to file uploads, API webhooks, database changes, and message queue events in real-time
- **Infrastructure Orchestration**: Manage CI/CD pipelines, Terraform runs, and Kubernetes deployments
- **Business Process Automation**: Automate approval workflows, notifications, and cross-system data synchronization

## Architecture Overview

Kestra uses a pluggable architecture:

```
┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│  YAML Flows │────▶│ Kestra Server│────▶│   Workers    │
│  (Git/UI)   │     │  (Executor)  │     │ (Task Runs)  │
└─────────────┘     └──────┬───────┘     └─────────────┘
                           │
                    ┌──────┴───────┐
                    │  Repository  │
                    │ (Postgres/   │
                    │  Elasticsearch)│
                    └──────────────┘
```

- **Flows**: Declarative YAML definitions with tasks, triggers, inputs, and outputs
- **Namespaces**: Organize flows into logical groups with shared variables and files
- **Triggers**: Schedule (cron), event-based (webhook, file detection, flow completion), or polling
- **Task Runners**: Execute tasks in Docker containers, Kubernetes pods, or cloud compute (AWS Batch, GCP, Azure)

## Key Features Deep Dive

### Declarative YAML Flows

```yaml
id: etl_pipeline
namespace: data.production

inputs:
  - id: date
    type: DATE
    defaults: "{{ now() | dateAdd(-1, 'DAYS') | date('yyyy-MM-dd') }}"

tasks:
  - id: extract
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: jdbc:postgresql://source-db:5432/analytics
    sql: SELECT * FROM events WHERE date = '{{ inputs.date }}'
    store: true

  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import pandas as pd
      df = pd.read_csv('{{ outputs.extract.uri }}')
      df['processed_at'] = pd.Timestamp.now()
      df.to_csv('{{ outputDir }}/transformed.csv', index=False)

  - id: load
    type: io.kestra.plugin.gcp.bigquery.Load
    from: "{{ outputs.transform.outputFiles['transformed.csv'] }}"
    destinationTable: project.dataset.events

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 2 * * *"
```

### Built-in Monitoring

Kestra provides real-time execution monitoring with Gantt charts, log streaming, topology views, and automatic failure alerts. The UI shows execution history, metrics, and allows manual replay of failed runs.

### Plugin Ecosystem

500+ official plugins covering:
- **Databases**: PostgreSQL, MySQL, MongoDB, ClickHouse, Snowflake, BigQuery
- **Cloud**: AWS (S3, Lambda, ECS), GCP (GCS, Dataproc), Azure (Blob, Functions)
- **Messaging**: Kafka, RabbitMQ, MQTT, Redis, Pulsar
- **Scripts**: Python, Node.js, R, Shell, PowerShell in isolated containers

## Self-Hosting Guide

### Docker Compose (Production)

```yaml
# docker-compose.yml
services:
  kestra:
    image: kestra/kestra:latest
    command: server standalone
    ports:
      - "8080:8080"
    volumes:
      - kestra-data:/app/storage
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: kestra
        kestra:
          repository:
            type: postgres
          queue:
            type: postgres
          storage:
            type: local
            local:
              base-path: /app/storage

  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: kestra
    volumes:
      - pg-data:/var/lib/postgresql/data

volumes:
  kestra-data:
  pg-data:
```

### Kubernetes with Helm

```bash
helm repo add kestra https://helm.kestra.io/
helm install kestra kestra/kestra --namespace kestra --create-namespace
```

## Kestra vs Alternatives

| Feature | Kestra | Airflow | Prefect | Temporal |
|---------|--------|---------|---------|----------|
| Configuration | YAML declarative | Python DAGs | Python decorators | Code-based |
| UI | Built-in visual editor | Web UI | Cloud dashboard | Web UI |
| Event-driven | Native | Limited | Yes | Yes |
| Learning curve | Low (YAML) | Medium (Python) | Medium | High |
| Plugin system | 500+ plugins | Operators | Integrations | Activities |
| Scaling | Horizontal | Celery/K8s | Cloud/K8s | Worker pools |

## FAQ

**Q: What team sizes is Kestra suitable for?**
A: Everything from solo developers to enterprise teams. Single-node Docker works for small-scale use; Kubernetes deployment supports millions of executions per day.

**Q: Do I need to know Java to use it?**
A: No. Flows are defined in YAML, and script tasks support Python, Node.js, Shell, and more. Java is only needed for developing custom plugins.

**Q: What are the advantages over Airflow?**
A: Kestra's declarative YAML approach is easier to pick up than Airflow's Python DAGs. It natively supports event-driven workflows, includes a visual editor, and lets you deploy new flows without restart.

## Source & Thanks

- GitHub: [kestra-io/kestra](https://github.com/kestra-io/kestra) — 26.7K+ ⭐ | Apache-2.0
- Website: [kestra.io](https://kestra.io)

---
Source: https://tokrepo.com/en/workflows/kestra-event-driven-orchestration-scheduling-platform-556ae291
Author: AI Open Source