Kestra — Event-Driven Orchestration & Scheduling Platform

What Kestra Does

Kestra handles workflow orchestration across multiple domains:

• Data Pipelines: Schedule and run ETL/ELT jobs with built-in support for dbt, Spark, Flink, and major databases
• Event-Driven Automation: React to file uploads, API webhooks, database changes, and message queue events in real-time
• Infrastructure Orchestration: Manage CI/CD pipelines, Terraform runs, and Kubernetes deployments
• Business Process Automation: Automate approval workflows, notifications, and cross-system data synchronization

Architecture Overview

Kestra uses a pluggable architecture:

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│  YAML Flows │────▶│ Kestra Server│────▶│   Workers    │
│  (Git/UI)   │     │  (Executor)  │     │ (Task Runs)  │
└─────────────┘     └──────┬───────┘     └─────────────┘
                           │
                    ┌──────┴───────┐
                    │  Repository  │
                    │ (Postgres/   │
                    │  Elasticsearch)│
                    └──────────────┘

• Flows: Declarative YAML definitions with tasks, triggers, inputs, and outputs
• Namespaces: Organize flows into logical groups with shared variables and files
• Triggers: Schedule (cron), event-based (webhook, file detection, flow completion), or polling
• Task Runners: Execute tasks in Docker containers, Kubernetes pods, or cloud compute (AWS Batch, GCP, Azure)

Key Features Deep Dive

Declarative YAML Flows

id: etl_pipeline
namespace: data.production

inputs:
  - id: date
    type: DATE
    defaults: "{{ now() | dateAdd(-1, 'DAYS') | date('yyyy-MM-dd') }}"

tasks:
  - id: extract
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: jdbc:postgresql://source-db:5432/analytics
    sql: SELECT * FROM events WHERE date = '{{ inputs.date }}'
    store: true

  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import pandas as pd
      df = pd.read_csv('{{ outputs.extract.uri }}')
      df['processed_at'] = pd.Timestamp.now()
      df.to_csv('{{ outputDir }}/transformed.csv', index=False)

  - id: load
    type: io.kestra.plugin.gcp.bigquery.Load
    from: "{{ outputs.transform.outputFiles['transformed.csv'] }}"
    destinationTable: project.dataset.events

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 2 * * *"

Built-in Monitoring

Kestra provides real-time execution monitoring with Gantt charts, log streaming, topology views, and automatic failure alerts. The UI shows execution history, metrics, and allows manual replay of failed runs.

Plugin Ecosystem

500+ official plugins covering:

• Databases: PostgreSQL, MySQL, MongoDB, ClickHouse, Snowflake, BigQuery
• Cloud: AWS (S3, Lambda, ECS), GCP (GCS, Dataproc), Azure (Blob, Functions)
• Messaging: Kafka, RabbitMQ, MQTT, Redis, Pulsar
• Scripts: Python, Node.js, R, Shell, PowerShell in isolated containers

Self-Hosting Guide

Docker Compose (Production)

# docker-compose.yml
services:
  kestra:
    image: kestra/kestra:latest
    command: server standalone
    ports:
      - "8080:8080"
    volumes:
      - kestra-data:/app/storage
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: kestra
        kestra:
          repository:
            type: postgres
          queue:
            type: postgres
          storage:
            type: local
            local:
              base-path: /app/storage

  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: kestra
    volumes:
      - pg-data:/var/lib/postgresql/data

volumes:
  kestra-data:
  pg-data:

Kubernetes with Helm

helm repo add kestra https://helm.kestra.io/
helm install kestra kestra/kestra --namespace kestra --create-namespace

Kestra vs Alternatives

FAQ

Q: What team sizes is Kestra suitable for? A: Everything from solo developers to enterprise teams. Single-node Docker works for small-scale use; Kubernetes deployment supports millions of executions per day.

Q: Do I need to know Java to use it? A: No. Flows are defined in YAML, and script tasks support Python, Node.js, Shell, and more. Java is only needed for developing custom plugins.

Q: What are the advantages over Airflow? A: Kestra's declarative YAML approach is easier to pick up than Airflow's Python DAGs. It natively supports event-driven workflows, includes a visual editor, and lets you deploy new flows without restart.