Apache Arrow — Columnar In-Memory Format and Compute Runtime

What it is

Apache Arrow defines a language-independent columnar memory layout for flat and hierarchical data. It provides libraries in C++, Java, Rust, Go, Python, R, JavaScript, and C# that read, write, transport, and process data without serialization overhead.

Arrow has become the common data plane for modern analytics tools. DuckDB, Polars, Pandas 2.x, Spark, Snowflake clients, and most contemporary data tools use Arrow as their interchange format, turning data-copying silos into a cooperative ecosystem.

How it saves time or tokens

Arrow eliminates serialization costs when moving data between tools. Without Arrow, transferring data from a Python process to a Spark job requires serializing to CSV or Parquet, writing to disk, and deserializing. With Arrow, data stays in memory in a shared format that both tools understand natively. Arrow Flight provides a gRPC-based protocol for moving Arrow batches over the network at near-memory speeds. The vectorized compute kernels (SIMD-accelerated filter, math, string, and aggregate operations) eliminate Python loops for data transformations.

How to use

1. Install PyArrow:

pip install 'pyarrow>=16'

1. Build a table, compute, and write to Parquet:

import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.parquet as pq

tbl = pa.table({'id': [1, 2, 3], 'price': [9.99, 19.5, 4.25]})
tbl = tbl.append_column('tax', pc.multiply(tbl['price'], 0.07))
tbl = tbl.filter(pc.greater(tbl['tax'], 0.5))
pq.write_table(tbl, '/tmp/orders.parquet')

1. Use zero-copy integration with Pandas:

import pandas as pd
df = tbl.to_pandas()  # zero-copy where possible
arrow_tbl = pa.Table.from_pandas(df)

Example

Using Arrow Flight to transfer data between services:

import pyarrow.flight as flight

# Client: connect and fetch data
client = flight.connect('grpc://data-service:8815')
info = client.get_flight_info(
    flight.FlightDescriptor.for_path('orders', '2026-04')
)
reader = client.do_get(info.endpoints[0].ticket)
table = reader.read_all()
print(f'Fetched {table.num_rows} rows in Arrow format')
# No serialization needed - data arrives as Arrow batches

Related on TokRepo

• Database tools — More data processing and database tools on TokRepo.
• Featured workflows — Discover curated analytics and developer tools.

Common pitfalls

• Assuming Arrow replaces Parquet. Arrow is an in-memory format; Parquet is a storage format. Use Arrow for computation and Parquet for persistence. They are complementary.
• Not using Arrow's compute kernels and falling back to Python loops defeats the performance benefit. Use pyarrow.compute for filtering, math, and string operations.
• Mixing Arrow versions across tools in the same pipeline can cause compatibility issues. Pin to the same major version of PyArrow across your data stack.