ast-grep — Structural Code Search and Rewrite Tool

Introduction

ast-grep (sg) is a code search and rewrite tool that operates on abstract syntax trees rather than raw text. By matching structural patterns instead of regular expressions, it finds and transforms code with syntactic precision, eliminating false positives from comments, strings, or formatting differences.

What ast-grep Does

• Searches codebases using AST patterns that match code structure, not string content
• Rewrites matched code with placeholder-based templates in a single command
• Supports rule files in YAML for building reusable lint checks and codemods
• Provides a language-server mode for real-time diagnostics in editors
• Covers JavaScript, TypeScript, Python, Rust, Go, C, Java, Kotlin, and more via tree-sitter

Architecture Overview

ast-grep is written in Rust and uses tree-sitter grammars to parse source files into concrete syntax trees. Pattern strings are themselves parsed into partial ASTs that are matched against the target code using a structural unification algorithm. Meta-variables like $VAR and $$$ARGS capture single nodes or variadic sequences, enabling flexible pattern matching. The engine walks the file tree in parallel for fast scanning of large repositories.

Self-Hosting & Configuration

• Install with cargo: cargo install ast-grep --locked
• Install with npm: npm install -g @ast-grep/cli
• Install with pip: pip install ast-grep-cli
• Create a sgconfig.yml at the project root to define custom lint rules
• Define YAML rule files under a rules/ directory for team-shared analysis

Key Features

• AST-based matching eliminates false positives from regex-based searches
• Variadic meta-variables ($$$) match zero or more arguments or statements
• YAML rule engine supports constraints like has, follows, and inside for complex patterns
• Built-in language server provides real-time linting and quick-fix actions
• Processes large codebases quickly due to Rust and parallel file walking

Comparison with Similar Tools

• ripgrep — regex text search; ast-grep understands code structure and avoids false matches
• Semgrep — similar AST-based scanning but server-dependent for some features; ast-grep is fully offline
• jscodeshift — JavaScript-only codemod tool; ast-grep supports many languages with a simpler pattern syntax
• comby — structural search tool; ast-grep uses real parsers via tree-sitter for higher accuracy
• ESLint custom rules — JS/TS only and requires plugin code; ast-grep rules are declarative YAML

FAQ

Q: Which languages does ast-grep support? A: JavaScript, TypeScript, Python, Rust, Go, C, C++, Java, Kotlin, Lua, and more via tree-sitter grammars.

Q: Can I use ast-grep as a linter in CI? A: Yes. Define rules in YAML and run sg scan in CI pipelines. It returns non-zero exit codes on violations.

Q: What are meta-variables? A: Placeholders like $X match a single AST node and $$$ARGS matches a sequence, similar to capture groups in regex but for syntax trees.

Q: Does it modify files in place? A: By default it prints diffs. Add --update-all to apply rewrites to files directly.

Sources

• https://github.com/ast-grep/ast-grep
• https://ast-grep.github.io