Noria — Streaming Dataflow SQL Engine for Web Applications

Introduction

Noria is a streaming dataflow system developed at MIT PDOS that compiles SQL queries into a dataflow graph of operators. As writes arrive, the operators incrementally maintain materialized views so that reads return pre-computed results with microsecond latency. It was designed to replace manual caching in read-heavy web applications.

What Noria Does

• Compiles SQL queries into a directed acyclic dataflow graph
• Incrementally maintains materialized views as new writes flow through the graph
• Serves reads from in-memory materialized state without hitting a backing database
• Supports joins, aggregations, and filters as streaming operators
• Adapts the dataflow graph dynamically when new queries are added at runtime

Architecture Overview

Noria consists of a controller and one or more worker processes. The controller accepts SQL query definitions and compiles them into a dataflow graph. Workers host the graph operators and maintain partial materialized state in memory. Writes enter the graph as update messages and propagate through operators that incrementally recompute affected rows. Reads are served from leaf nodes that hold the final materialized results. Partial state eviction keeps memory usage bounded under workloads where not all keys are hot.

Self-Hosting & Configuration

• Clone the repository and build with Cargo; requires a recent stable Rust toolchain
• Launch noria-server with a deployment name and optional ZooKeeper address for multi-worker setups
• Define base tables and queries through the Noria client API or a MySQL-protocol adapter
• Configure memory limits and eviction policies for partial state management
• Use the MySQL adapter to connect existing applications without driver changes

Key Features

• Microsecond read latency from pre-computed materialized views
• Automatic incremental maintenance — no manual cache invalidation
• Dynamic query addition without restarting the system
• Partial state eviction to keep memory usage bounded
• MySQL-compatible protocol adapter for drop-in integration

Comparison with Similar Tools

• ReadySet — commercial product built on Noria's research; Noria is the original open-source prototype
• Materialize — full streaming SQL database; Noria focuses on web-application read paths
• Redis — manual key-value cache; Noria automates cache maintenance for SQL results
• Apache Flink — general stream processing; Noria is optimized for low-latency view maintenance
• ClickHouse — analytical OLAP engine; Noria targets transactional read-heavy workloads

FAQ

Q: Is Noria production-ready? A: Noria is a research prototype. It demonstrates the streaming dataflow concept and has been the basis for commercial systems like ReadySet, but it is not packaged for production deployment.

Q: How does Noria differ from a traditional materialized view? A: Traditional materialized views are refreshed periodically or on demand. Noria maintains views incrementally and continuously as writes arrive, so reads always see fresh data.

Q: Can Noria handle joins across multiple tables? A: Yes. Joins are compiled into dataflow operators that maintain state for both sides and emit updates when either input changes.

Q: What happens when a worker fails? A: The controller can redistribute graph operators to surviving workers. Partial state is rebuilt from base tables as needed.

Sources

• https://github.com/mit-pdos/noria
• https://pdos.csail.mit.edu/papers/noria:osdi18.pdf