Kestra — Event-Driven Orchestration & Scheduling Platform

What it is

Kestra is an open-source orchestration platform for scheduling and running complex data pipelines, ETL jobs, and automation workflows. You define workflows in declarative YAML, and Kestra handles scheduling, dependencies, retries, and monitoring. It supports event-driven triggers, API triggers, and cron-based scheduling.

This tool is for data engineers, DevOps engineers, and platform teams who need workflow orchestration without the complexity of Airflow or the limitations of cron. It provides a web UI for monitoring and managing workflows.

How it saves time or tokens

Kestra's declarative YAML approach means no Python code required for defining workflows, unlike Airflow. The built-in plugin system supports 400+ integrations (databases, cloud services, messaging) without custom code. The web UI provides real-time monitoring, log viewing, and manual workflow triggering, reducing the need for separate monitoring tools.

How to use

1. Deploy Kestra via Docker or Kubernetes.
2. Define workflows in YAML files.
3. Upload workflows through the UI or API.
4. Monitor executions in the dashboard.

# Start Kestra with Docker
docker run --pull=always --rm -it -p 8080:8080 \
  --user=root \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp:/tmp \
  kestra/kestra:latest server local

# Access UI at http://localhost:8080

Example

A data pipeline workflow:

id: daily_etl
namespace: production

triggers:
  - id: schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: '0 6 * * *'

tasks:
  - id: extract
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: jdbc:postgresql://db:5432/source
    sql: SELECT * FROM orders WHERE date = '{{ trigger.date }}'
    store: true

  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import pandas as pd
      df = pd.read_csv('{{ outputs.extract.uri }}')
      df['total'] = df['quantity'] * df['price']
      df.to_csv('{{ outputDir }}/transformed.csv')

  - id: load
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: jdbc:postgresql://db:5432/warehouse
    sql: COPY orders_fact FROM '{{ outputs.transform.uri }}'

Related on TokRepo

• Automation tools — More workflow automation platforms
• DevOps tools — Infrastructure orchestration

Common pitfalls

• YAML indentation errors cause silent failures. Validate your workflow files before uploading.
• The plugin system is powerful but each plugin adds dependencies. Keep your Kestra instance lean by only installing plugins you use.
• Complex conditional logic is harder to express in YAML than in Python. For heavy transformation logic, use script tasks.
• Kestra's learning curve includes understanding namespaces, flows, triggers, and the execution model.
• Self-hosted Kestra requires PostgreSQL for production. The embedded H2 database is for testing only.
• Review the official documentation before deploying to production to ensure compatibility with your specific environment and requirements.