xan — Blazing-Fast CSV Swiss Army Knife for the Command Line

Introduction

xan addresses the pain of working with CSV data in the terminal by providing a single tool that replaces fragmented workflows of awk, cut, grep, and Python scripts. Built in Rust for speed, it processes multi-gigabyte files efficiently while offering an expression language that makes complex column transformations and filtering straightforward.

What xan Does

• Selects, renames, and reorders columns with a concise syntax
• Filters rows using regex patterns or a built-in expression language
• Computes statistics, frequency tables, and histograms over columns
• Joins, merges, and deduplicates CSV files by key columns
• Handles large files efficiently with streaming processing and minimal memory usage

Architecture Overview

xan is implemented in Rust using the csv crate for fast parsing and a custom expression evaluator for the transformation language. Commands are structured as subcommands (select, filter, map, stats) that compose via Unix pipes. The parser processes CSV records in a streaming fashion, avoiding loading entire files into memory. The expression language supports string operations, arithmetic, conditionals, and regex matching, compiled to an internal representation for evaluation speed.

Self-Hosting & Configuration

• Install via cargo install xan or download pre-built binaries from GitHub releases
• No configuration files needed; all options are passed via command-line flags
• Set default delimiter with the -d flag for TSV or custom-delimited files
• Pipe multiple xan commands together for complex multi-step transformations
• Use --no-headers flag for headerless files and --output to write results to a file

Key Features

• Built-in expression language with string, math, and regex functions for column transformations
• Streaming architecture handles multi-gigabyte files without loading them into memory
• Frequency analysis and descriptive statistics computed in a single pass
• Join operations across multiple CSV files by key columns
• Colorized terminal output for better readability of results

Comparison with Similar Tools

• xsv — the predecessor project; xan adds an expression language, more commands, and active maintenance
• Miller (mlr) — supports multiple formats (JSON, CSV); xan is CSV-focused with a richer expression syntax
• csvkit — Python-based CSV toolkit; xan offers significantly faster processing for large files
• q — runs SQL on CSV files; xan provides a purpose-built expression language instead of SQL

FAQ

Q: How fast is xan compared to Python-based tools? A: xan processes CSV files orders of magnitude faster than Python tools thanks to Rust performance and streaming I/O.

Q: Does xan support TSV and other delimiters? A: Yes. Use the -d flag to specify any delimiter character.

Q: Can I chain xan commands together? A: Yes. xan commands compose naturally via Unix pipes, similar to standard Unix text utilities.

Q: Does xan modify files in place? A: No. xan reads from stdin or a file and writes to stdout or a specified output file, leaving the original unchanged.

Sources

• https://github.com/medialab/xan