# CloudQuery — Sync Cloud Infrastructure to SQL for Security and Compliance

> CloudQuery is an open-source ELT framework that extracts configuration data from cloud APIs, SaaS platforms, and databases into PostgreSQL or data lakes for security, compliance, and asset visibility.

## Install

Save as a script file and run:

# CloudQuery — Sync Cloud Infrastructure to SQL for Security and Compliance

## Quick Use
```bash
# Install CloudQuery CLI
brew install cloudquery/tap/cloudquery

# Initialize a config file for AWS to PostgreSQL
cloudquery login
cat > cloudquery.yml <<EOF
kind: source
spec:
  name: aws
  path: cloudquery/aws
  tables: ["aws_s3_buckets", "aws_ec2_instances"]
---
kind: destination
spec:
  name: postgresql
  path: cloudquery/postgresql
  spec:
    connection_string: "postgresql://localhost:5432/cloudquery"
EOF

# Run the sync
cloudquery sync cloudquery.yml
```

## Introduction
CloudQuery is an open-source ELT (Extract, Load, Transform) framework purpose-built for infrastructure and security data. It connects to cloud providers (AWS, GCP, Azure), SaaS tools (GitHub, Okta, Cloudflare), and other APIs through a plugin system, extracts their configuration and resource data, and loads it into SQL databases or data lakes. Security and platform teams use CloudQuery to build asset inventories, run compliance queries, detect drift, and feed SIEM pipelines.

## What CloudQuery Does
- Syncs cloud resource configurations from 100+ sources into PostgreSQL, BigQuery, S3, or Snowflake
- Provides pre-built source plugins for AWS, GCP, Azure, Kubernetes, GitHub, Okta, and many more
- Enables SQL-based security queries like finding public S3 buckets or unencrypted volumes
- Supports incremental syncs so only changed resources are updated on subsequent runs
- Ships with policy packs for CIS benchmarks and compliance frameworks

## Architecture Overview
CloudQuery uses a plugin-based architecture with source plugins and destination plugins connected by a high-performance gRPC streaming protocol. Each source plugin authenticates with a cloud API, fetches resources according to the configured table list, and streams rows to the destination plugin which writes them to the target database. Plugins run as separate processes, enabling language-agnostic development. The CLI orchestrates plugin lifecycle, schema migrations, and sync scheduling.

## Self-Hosting & Configuration
- Install the cloudquery CLI via Homebrew, package managers, or download from GitHub releases
- Create a YAML config file defining source plugins (cloud providers) and destination plugins (databases)
- Configure authentication using standard cloud credentials (AWS profiles, GCP service accounts)
- Select specific tables to sync for faster, targeted data extraction
- Schedule syncs with cron or integrate into CI/CD pipelines for continuous asset inventory

## Key Features
- 100+ source plugins covering major cloud providers, SaaS platforms, and infrastructure tools
- High-performance Go-based sync engine that handles millions of resources efficiently
- SQL-native approach lets you query infrastructure data with standard SQL joins and aggregations
- Pre-built compliance and security policy packs for CIS AWS, GCP, and Azure benchmarks
- Extensible plugin SDK for building custom source or destination plugins in Go or Python

## Comparison with Similar Tools
- **Steampipe** — SQL-based cloud querying with live API calls per query, while CloudQuery syncs data to a persistent database for faster repeated queries
- **Prowler** — focused on security checks and compliance scoring, not general-purpose asset inventory
- **AWS Config** — native AWS service for resource tracking but limited to AWS and expensive at scale
- **Cartography (Lyft)** — Neo4j-based infrastructure graph but narrower source coverage and less active
- **Resoto (Some Engineering)** — cloud asset inventory with graph model but smaller community and plugin ecosystem

## FAQ
**Q: How often should I run CloudQuery syncs?**
A: Most teams run syncs every 1-4 hours for security use cases. For compliance snapshots, daily or weekly syncs are common. Incremental mode reduces sync time significantly.

**Q: Does CloudQuery support multi-cloud environments?**
A: Yes. Configure multiple source plugins in the same config file. AWS, GCP, and Azure resources land in the same database, enabling cross-cloud SQL queries.

**Q: Can I write custom source plugins?**
A: Yes. CloudQuery provides an SDK for building source plugins in Go or Python. The plugin communicates with the CLI via gRPC and follows a standard table/column schema.

**Q: What databases does CloudQuery support as destinations?**
A: PostgreSQL, BigQuery, Snowflake, S3 (Parquet/CSV), ClickHouse, Elasticsearch, Apache Kafka, and more via destination plugins.

## Sources
- https://github.com/cloudquery/cloudquery
- https://www.cloudquery.io/docs

---
Source: https://tokrepo.com/en/workflows/a36fe8b8-3987-11f1-9bc6-00163e2b0d79
Author: Script Depot