Apache DataFusion — Fast In-Process SQL Query Engine in Rust

What it is

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory columnar format. It provides a SQL interface for running analytical queries directly inside your application, without a separate database server. DataFusion handles query parsing, optimization, and execution with support for Parquet, CSV, JSON, and Avro file formats.

DataFusion targets developers building data-intensive applications in Rust who need SQL capabilities without the overhead of an external database. It suits embedded analytics, data lake query engines, and custom database products.

How it saves time or tokens

This workflow provides the Cargo dependency and a working Rust example that reads a CSV file and runs SQL queries. Instead of setting up a database server for analytical queries, you add a single crate to your Cargo.toml and query data files directly.

How to use

1. Add DataFusion to your Cargo.toml:

cargo add datafusion tokio

1. Write a query:

use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();
    ctx.register_csv("sales", "sales.csv", CsvReadOptions::new()).await?;

    let df = ctx.sql("SELECT category, SUM(amount) as total FROM sales GROUP BY category ORDER BY total DESC").await?;
    df.show().await?;
    Ok(())
}

1. Run your application:

cargo run

Example

use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();

    // Register a Parquet file as a table
    ctx.register_parquet("logs", "access_logs.parquet", ParquetReadOptions::default()).await?;

    // Run analytical queries
    let top_paths = ctx.sql("
        SELECT path, COUNT(*) as hits, AVG(response_time_ms) as avg_rt
        FROM logs
        WHERE status_code = 200
        GROUP BY path
        ORDER BY hits DESC
        LIMIT 10
    ").await?;

    top_paths.show().await?;
    Ok(())
}

Related on TokRepo

• Database tools -- Data processing and query engines
• AI tools for research -- Analytical tools for data research

Common pitfalls

• DataFusion is an in-process engine, not a database server. It does not persist data or manage transactions. Use it for analytical queries on files or for building custom database products.
• Large datasets that exceed available memory cause OOM errors. DataFusion streams results but still needs memory for intermediate aggregations.
• The async API requires Tokio runtime. Make sure your application uses #[tokio::main] or creates a Tokio runtime explicitly.