# data.table — High-Performance Data Manipulation for R

> data.table is an R package that extends data.frame with fast aggregation, ordered joins, and in-place modification by reference. It provides a concise syntax for filtering, grouping, and reshaping large datasets with minimal memory overhead.

## Install

Save as a script file and run:

# data.table — High-Performance Data Manipulation for R

## Quick Use
```r
install.packages("data.table")
library(data.table)
dt <- fread("flights.csv")
dt[origin == "JFK" & month == 6, .(avg_delay = mean(arr_delay, na.rm=TRUE)), by = carrier][order(-avg_delay)]
```

## Introduction
data.table is an R package created by Matt Dowle that provides a high-performance alternative to base R data frames. Its concise `DT[i, j, by]` syntax covers filtering, column operations, and grouping in a single expression. It is widely used in finance, genomics, and other fields that work with datasets of tens or hundreds of millions of rows.

## What data.table Does
- Reads and writes CSV files at high speed with `fread()` and `fwrite()`, auto-detecting types and delimiters
- Filters rows (i), computes or selects columns (j), and groups (by) in a single `DT[i, j, by]` expression
- Updates columns in place by reference with `:=`, avoiding memory copies
- Performs fast ordered joins, rolling joins, and non-equi joins for time-series and interval data
- Reshapes data between long and wide formats with `melt()` and `dcast()`

## Architecture Overview
data.table stores data as a list of column vectors with an internal row-order index. The `[.data.table` method parses the i, j, and by expressions to generate an optimized execution plan. Grouping uses radix-based sorting and hashing. Joins leverage automatic key-based binary search. Modification by reference updates column pointers without copying the entire table, which is critical for large datasets. The package is written in C for core operations and uses OpenMP for parallelism where available.

## Self-Hosting & Configuration
- Install from CRAN with `install.packages("data.table")`
- Enable multi-threading with `setDTthreads(0)` (uses all available cores) or set a specific count
- Set keys for fast binary-search-based subsetting: `setkey(dt, column)`
- Use `fread()` options like `select`, `nrows`, and `key` for optimized file reading
- Integrate with ggplot2 by passing data.table objects directly (they inherit from data.frame)

## Key Features
- Concise `DT[i, j, by]` syntax replaces verbose chains of subset/transform/aggregate calls
- Modification by reference with `:=` avoids expensive copies and reduces peak memory use
- Automatic indexing and key-based binary search make row filtering extremely fast
- Rolling and overlap joins handle time-series alignment and interval queries natively
- `fread()` and `fwrite()` are among the fastest CSV readers/writers in any language

## Comparison with Similar Tools
- **dplyr (R)** — more readable verb-based API but slower on very large datasets; data.table trades readability for speed
- **pandas (Python)** — similar feature set but data.table is often faster for grouped aggregations on large data; pandas uses more memory
- **Polars (Rust/Python)** — newer columnar library with comparable speed; data.table has deeper R integration and a longer track record
- **Arrow / DuckDB** — out-of-core analytics engines; data.table excels at in-memory operations up to available RAM
- **SQL** — data.table's syntax maps loosely to SQL; it is best for in-process analysis without a database server

## FAQ
**Q: When should I choose data.table over dplyr?**
A: When working with large datasets (millions of rows), when memory efficiency matters, or when you need rolling/non-equi joins. For smaller data and readability, dplyr may be simpler.

**Q: Does data.table work with tidyverse packages?**
A: Yes. data.table inherits from data.frame, so it works with ggplot2, tidyr, and most tidyverse functions. The dtplyr package translates dplyr syntax to data.table operations.

**Q: How fast is fread compared to read.csv?**
A: fread is typically 10-50x faster than base R's read.csv and comparable to or faster than Python's pandas.read_csv for large files.

**Q: Can data.table handle billions of rows?**
A: data.table works in memory, so it is limited by available RAM. For datasets exceeding RAM, consider Arrow, DuckDB, or database backends.

## Sources
- https://github.com/Rdatatable/data.table
- https://rdatatable.gitlab.io/data.table/

---
Source: https://tokrepo.com/en/workflows/asset-477c0f01
Author: Script Depot