Pathway — Python ETL Framework for Stream Processing and RAG

Introduction

Pathway is a Python data processing framework that handles both batch and streaming workloads with a single API. It provides a Rust-powered engine underneath Python syntax, making it suitable for building real-time analytics, ETL pipelines, and LLM-powered applications like RAG without switching tools.

What Pathway Does

• Processes live data streams and static datasets with the same Python code
• Provides a DataFrame-like API backed by a high-performance Rust engine
• Includes built-in connectors for Kafka, S3, PostgreSQL, and file systems
• Supports incremental computation so only changed data is reprocessed
• Offers native integration with LLM frameworks for RAG and document indexing

Architecture Overview

Pathway's core is a Rust-based dataflow engine that executes Python-defined computation graphs. When you write transformations using the Python API, they are compiled into an optimized execution plan that handles both bounded (batch) and unbounded (streaming) data. The engine tracks data changes incrementally, re-computing only affected parts of the pipeline when new data arrives.

Self-Hosting & Configuration

• Install via pip with pip install pathway on Linux, macOS, or WSL
• Define data sources using pw.io connectors for Kafka, S3, CSV, JSON, or databases
• Configure persistence by setting checkpoint directories for fault tolerance
• Deploy as a standalone Python process, Docker container, or on Kubernetes
• Set environment variables for connector credentials and engine tuning parameters

Key Features

• Unified batch and streaming: write once, run on static files or live streams
• Incremental computation engine avoids full reprocessing on each update
• LLM toolbox with vector indexing, chunking, and embedding support for RAG
• Over 350 built-in data transformations including joins, aggregations, and windowing
• Interoperability with pandas, NumPy, and existing Python data science libraries

Comparison with Similar Tools

• Apache Flink — JVM-based, heavier operational overhead; Pathway runs as a Python process
• Apache Spark Structured Streaming — requires a cluster; Pathway runs on a single machine or scales out
• Bytewax — also Python-native streaming, but Pathway has broader connector ecosystem and LLM integrations
• Kafka Streams — Java/Kotlin only; Pathway provides the same streaming semantics in Python

FAQ

Q: Does Pathway require a cluster or external infrastructure? A: No. It runs as a single Python process for development and can scale horizontally for production workloads.

Q: Can Pathway replace my existing batch ETL pipeline? A: Yes. The same code that processes static files can switch to streaming mode by changing the input connector, with no code modifications.

Q: How does Pathway integrate with LLMs for RAG? A: It includes built-in operators for document parsing, chunking, embedding, and vector indexing that update incrementally as source documents change.

Q: What data formats and sources does Pathway support? A: CSV, JSON, Parquet, Kafka, S3, PostgreSQL, MongoDB, Google Drive, SharePoint, and more via its connector library.

Sources

• https://github.com/pathwaycom/pathway
• https://pathway.com/developers/documentation