Daft — High-Performance Data Engine for AI Workloads

Introduction

Daft is a DataFrame engine built for AI and multimodal data processing. Unlike traditional DataFrame libraries that focus on tabular data, Daft natively supports columns of images, embeddings, audio, and video alongside standard numeric and string types.

What Daft Does

• Provides a lazy DataFrame API for expressing complex data transformations
• Processes multimodal columns (images, audio, tensors) with first-class support
• Reads and writes Parquet, CSV, JSON, Apache Iceberg, Delta Lake, and Hudi formats
• Scales from single-machine execution to distributed Ray clusters without code changes
• Integrates with ML frameworks for embedding generation and model inference within pipelines

Architecture Overview

Daft uses a Rust-based query planner and execution engine for performance. Queries are lazily constructed as logical plans and optimized before execution. The engine supports both local multi-threaded execution and distributed execution on Ray. Data is processed in Apache Arrow format for zero-copy interop with other tools.

Self-Hosting & Configuration

• Install via pip with optional extras for Ray, AWS, or visualization support
• Use the DataFrame API to read from local files, cloud storage, or catalog tables
• Configure Ray for distributed execution by pointing to an existing Ray cluster
• Set memory limits and partition sizes to tune performance for your workload
• Enable native Iceberg or Delta Lake catalog integration for lakehouse queries

Key Features

• Native multimodal data types for images, embeddings, audio, and tensors
• Rust-powered execution engine with Apache Arrow memory format
• Seamless scaling from laptop to distributed Ray cluster
• Built-in UDF support for running Python or ML model inference per partition
• SQL query interface alongside the Python DataFrame API

Comparison with Similar Tools

• Polars — Rust-based DataFrame library focused on tabular data; Daft adds multimodal and distributed support
• PySpark — Mature distributed engine but heavyweight and JVM-based; Daft is lighter with a Rust core
• DuckDB — Excellent for analytical SQL on a single machine; Daft targets distributed multimodal workloads
• Modin — pandas-compatible distributed DataFrame; Daft offers a purpose-built API for AI pipelines
• Vaex — Out-of-core DataFrame library; Daft provides richer distributed execution and multimodal types

FAQ

Q: Can Daft replace pandas? A: Daft is not a pandas drop-in replacement. It offers its own API optimized for lazy evaluation and multimodal data.

Q: Does Daft require Ray for distributed execution? A: No. Daft runs locally by default and only requires Ray when you need to scale across multiple machines.

Q: What file formats does Daft support? A: Parquet, CSV, JSON, Apache Iceberg, Delta Lake, and Hudi, with more formats planned.

Q: Is Daft suitable for traditional BI workloads? A: It can handle them, but tools like DuckDB or Polars may be more appropriate for pure tabular analytics.

Sources

• https://github.com/Eventual-Inc/Daft
• https://www.getdaft.io/projects/docs