jq — Lightweight Command-Line JSON Processor

Introduction

jq is like sed for JSON data — a command-line tool that lets you extract, transform, and manipulate JSON with a powerful expression language. In an era where every API returns JSON, jq has become an essential tool for developers, DevOps engineers, and data analysts working in the terminal.

With over 34,000 GitHub stars, jq is installed on millions of machines and is referenced in countless tutorials, scripts, and CI/CD pipelines. Its expression language is concise yet powerful enough to handle complex data transformations.

What jq Does

jq reads JSON input (from stdin, files, or pipes), applies transformation expressions, and outputs the result. It can extract fields, filter arrays, reshape objects, perform calculations, and format output — all without writing a full script in Python or JavaScript.

Architecture Overview

[Input JSON]
stdin, file, or pipe
        |
   [jq Expression]
   Filters, selects,
   transforms data
        |
   [jq Parser & Evaluator]
   Lexer -> Parser -> Bytecode
   -> Stack-based VM
        |
[Output]
Formatted JSON,
raw text, or TSV

[Expression Types]
.field          Select field
.[]             Iterate array
| pipe           Chain filters
select(cond)    Filter
map(expr)       Transform array
group_by(.key)  Group
sort_by(.key)   Sort

Self-Hosting & Configuration

# Common jq recipes

# Pretty-print a file
jq . data.json

# Extract nested fields
echo '{"user":{"name":"Alice","address":{"city":"NYC"}}}' | jq .user.address.city

# Filter array elements
echo '[{"name":"Alice","age":30},{"name":"Bob","age":25}]' | jq '.[] | select(.age > 27)'

# Transform array
echo '[{"name":"Alice","score":95},{"name":"Bob","score":87}]' | jq '[.[] | {student: .name, grade: .score}]'

# Count items
echo '[1,2,3,4,5]' | jq length

# Unique values
echo '["a","b","a","c","b"]' | jq unique

# Group and aggregate
cat sales.json | jq 'group_by(.region) | map({region: .[0].region, total: map(.amount) | add})'

# Convert JSON to CSV
echo '[{"name":"Alice","age":30},{"name":"Bob","age":25}]' | jq -r '.[] | [.name, .age] | @csv'

# Merge JSON objects
jq -s '.[0] * .[1]' defaults.json overrides.json

# Process API pagination
curl -s "https://api.example.com/items?page=1" | jq '.items[] | {id, name, status}'

Key Features

• JSON Parsing — read and pretty-print any JSON data
• Field Selection — extract nested values with dot notation
• Array Operations — filter, map, sort, group, flatten, and unique
• Object Manipulation — construct, merge, and reshape objects
• String Functions — split, join, test (regex), sub, gsub
• Math Operations — add, min, max, floor, ceil, and more
• Output Formats — JSON, raw text, CSV via @csv, TSV via @tsv
• Streaming — process large files without loading entirely into memory

Comparison with Similar Tools

FAQ

Q: How do I learn jq expressions? A: Start with ".field" for extraction, ".[]" for array iteration, and "| pipe" for chaining. The jq playground (jqplay.org) lets you experiment interactively.

Q: Can jq handle large files? A: Yes. Use --stream flag for very large files to process them incrementally without loading everything into memory.

Q: How do I use jq in shell scripts? A: Use -r for raw output (no quotes): result=$(echo $json | jq -r .name). Use -e for exit codes based on output (useful in conditionals).

Q: What is yq? A: yq is "jq for YAML" — it uses the same expression language but works with YAML files. Essential for processing Kubernetes manifests and Docker Compose files.

Sources

• GitHub: https://github.com/jqlang/jq
• Documentation: https://jqlang.github.io/jq/manual
• Playground: https://jqplay.org
• Created by Stephen Dolan
• License: MIT