Steampipe — Query Cloud APIs Live with SQL

Introduction

Steampipe exposes cloud and SaaS APIs as live PostgreSQL tables using a custom FDW. Instead of stitching together AWS CLI, kubectl, and GitHub scripts, operators run SQL joins across providers to answer audit and inventory questions in seconds.

What Steampipe Does

• Installs plugins that map API calls to Postgres tables
• Caches responses to keep queries cheap and rate-limit-friendly
• Ships Mods (packaged queries + benchmarks) for CIS, PCI, HIPAA
• Powers dashboards defined in HCL that render in a local browser
• Connects to BI tools like Metabase or Superset via Postgres wire protocol

Architecture Overview

Steampipe embeds PostgreSQL with a custom foreign data wrapper. Each plugin is a Go binary that translates SQL predicates into API calls, returns rows, and caches results in memory. A steampipe service mode keeps the DB hot so external clients can connect on port 9193.

Self-Hosting & Configuration

• macOS, Linux, Windows, Docker images available
• Plugins via steampipe plugin install <name> from the hub
• Auth reuses existing cloud credentials (AWS profiles, kubeconfig, GH token)
• Connection files aggregate multiple accounts/regions behind one schema
• steampipe service start exposes Postgres for Grafana/Tableau/DBeaver

Key Features

• Zero-ETL — queries hit APIs live, no warehouse to sync
• 140+ plugins: AWS, Azure, GCP, Kubernetes, GitHub, Okta, Slack, and more
• SQL JOINs across providers (e.g., AWS IAM ↔ Okta users)
• Prebuilt compliance mods for CIS/NIST benchmarks
• AGPL-3 core with permissive plugin licenses

Comparison with Similar Tools

• CloudQuery — ETL model, stores snapshots in Postgres/BigQuery
• Scout Suite — static audit report generator
• Prowler — CLI audit script for AWS/Azure/GCP
• osquery — host-level SQL, not cloud APIs
• CloudCustodian — policy language, more prescriptive

FAQ

Q: Does it store my data? A: Only in the embedded Postgres' cache; it is ephemeral unless you enable CloudQuery-style mods.

Q: How are credentials handled? A: Plugins reuse the same SDK chains as their native CLIs — profiles, env vars, IRSA, etc.

Q: Can I write custom plugins? A: Yes — the plugin SDK is Go and well-documented.

Q: How does it fit in CI? A: Run steampipe check all against a compliance mod and fail on non-zero exit.

Sources

• https://github.com/turbot/steampipe
• https://steampipe.io/docs