# Apache DataFusion — Fast In-Process SQL Query Engine in Rust

> An extensible query engine written in Rust that uses Apache Arrow as its in-memory format, enabling fast analytical SQL queries embeddable in any application.

## Install

Save as a script file and run:

# Apache DataFusion — Fast In-Process SQL Query Engine in Rust

## Quick Use
```bash
cargo add datafusion tokio
```
```rust
use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();
    ctx.register_csv("sales", "sales.csv", CsvReadOptions::new()).await?;
    let df = ctx.sql("SELECT region, SUM(amount) FROM sales GROUP BY region").await?;
    df.show().await?;
    Ok(())
}
```

## Introduction
Apache DataFusion is a fast, extensible query engine written in Rust that uses Apache Arrow's columnar memory format. It can be embedded directly into applications to provide SQL query capabilities over CSV, Parquet, JSON, and custom data sources without requiring a separate database server.

## What DataFusion Does
- Executes SQL and DataFrame queries against local files and custom data sources
- Uses Apache Arrow columnar format for vectorized, cache-friendly processing
- Supports CSV, Parquet, JSON, and Avro file formats out of the box
- Provides a DataFrame API alongside SQL for programmatic query construction
- Embeds as a library in Rust, Python (via datafusion-python), and other languages

## Architecture Overview
DataFusion processes queries through a pipeline: SQL parsing, logical planning, optimization, physical planning, and execution. The optimizer applies rule-based and cost-based transformations like predicate pushdown, projection pruning, and join reordering. Physical operators use Arrow RecordBatches for vectorized execution across partitioned data. The catalog system is pluggable, allowing custom table providers for any data source.

## Self-Hosting & Configuration
- Add datafusion as a Cargo dependency for Rust projects or pip install datafusion for Python
- Create a SessionContext and register data sources (files, tables, or custom providers)
- Configure memory limits, target partition count, and batch size via SessionConfig
- Implement custom TableProvider trait to query proprietary data formats
- Use the datafusion-cli binary for interactive SQL querying of local files

## Key Features
- Vectorized execution engine leveraging Arrow SIMD operations for analytical workloads
- Extensible optimizer with pluggable rules for domain-specific query transformations
- User-defined functions (UDFs), aggregates (UDAFs), and window functions
- Partitioned parallel execution across CPU cores with configurable concurrency
- Foundation for projects like Ballista (distributed query), InfluxDB IOx, and Delta Lake

## Comparison with Similar Tools
- **DuckDB** — similar in-process analytics; DataFusion is Rust-native and more extensible as a library
- **Apache Spark** — distributed big data engine; DataFusion is single-node, embeddable, and lighter
- **Polars** — DataFrame library in Rust; DataFusion adds full SQL support and query optimization
- **SQLite** — embedded relational DB; DataFusion is columnar and optimized for analytics over row storage
- **Velox** — Meta's C++ execution engine; DataFusion is Rust-based with a complete SQL frontend

## FAQ
**Q: Is DataFusion a database?**
A: No. It is a query engine library. It does not manage storage or provide a server. Projects like InfluxDB IOx build databases on top of DataFusion.

**Q: How does DataFusion compare to DuckDB?**
A: Both excel at in-process analytics. DuckDB is a complete embedded database. DataFusion is a query engine framework designed to be extended and embedded into larger systems.

**Q: Can I use DataFusion from Python?**
A: Yes. The datafusion-python package provides a Python API with both SQL and DataFrame interfaces backed by the Rust engine.

**Q: What is Ballista?**
A: Ballista is a distributed query engine built on DataFusion that spreads execution across multiple nodes using Arrow Flight for data transfer.

## Sources
- https://github.com/apache/datafusion
- https://datafusion.apache.org

---
Source: https://tokrepo.com/en/workflows/c39e1e76-39e1-11f1-9bc6-00163e2b0d79
Author: Script Depot