Pipedream — Code-First Workflow Automation Platform

Introduction

Pipedream is a code-first integration and workflow automation platform. It provides event-driven triggers, pre-built connectors for thousands of APIs, and the ability to write custom workflow steps in Node.js, Python, Go, or Bash. The platform is partially open source, with its component registry and SDK available on GitHub.

What Pipedream Does

• Triggers workflows from HTTP webhooks, cron schedules, email, or events from connected services
• Executes code steps in Node.js, Python, Go, or Bash with full standard library access
• Provides pre-built actions and triggers for services like GitHub, Slack, Stripe, Google Sheets, and more
• Manages authentication and OAuth tokens for connected APIs automatically
• Stores workflow state and provides key-value data stores for persisting data between runs

Architecture Overview

Pipedream workflows are serverless pipelines that execute in a managed runtime. Each workflow consists of a trigger (the event source) and one or more steps (code or pre-built actions). The open-source component registry on GitHub contains the source code for all triggers and actions, which are versioned Node.js/Python modules. The CLI and SDK allow local development and deployment of components. Event processing, scheduling, and execution infrastructure are managed by the Pipedream platform.

Self-Hosting & Configuration

• Use the CLI to develop and test workflow steps locally before deploying
• Components in the open-source registry can be forked and customized for specific needs
• Configure triggers via the web UI or YAML definitions with the CLI
• Set environment variables and secrets in the Pipedream dashboard, accessible in code steps
• Use the SDK to build custom event sources that poll APIs or listen for webhooks

Key Features

• Write workflow logic in real code (Node.js, Python, Go, Bash) rather than drag-and-drop only
• Component registry with source code for 2000+ API integrations on GitHub
• Built-in OAuth handling that manages token refresh for connected accounts
• Event-driven architecture with HTTP, cron, email, and app-specific triggers
• CLI and SDK for local development, testing, and deployment of components

Comparison with Similar Tools

• n8n — Self-hosted visual automation; Pipedream is code-first with a managed runtime option
• Zapier — No-code automation with proprietary integrations; Pipedream offers open-source components and code steps
• Activepieces — Self-hosted Zapier alternative; Pipedream provides a richer code-first developer experience
• Temporal — Durable workflow engine for application logic; Pipedream focuses on API integrations and event-driven automation

FAQ

Q: Is Pipedream fully open source? A: The component registry, CLI, and SDK are open source on GitHub. The execution platform and web UI are managed services.

Q: Can I self-host Pipedream? A: The execution runtime is not self-hostable. You can use the open-source components in your own Node.js/Python applications outside of Pipedream.

Q: What languages can I use in workflow steps? A: Node.js, Python, Go, and Bash are all supported as step languages within a single workflow.

Q: Is there a free tier? A: Yes. Pipedream offers a free tier with a daily invocation limit. Paid plans increase limits and add features like longer execution time.

Sources

• https://github.com/PipedreamHQ/pipedream
• https://pipedream.com/docs