Tree-sitter — Incremental Parser Generator for Editors

What it is

Tree-sitter is a parser generator tool and an incremental parsing library. It builds concrete syntax trees for source code files and updates them efficiently as you edit. Tree-sitter powers syntax highlighting, code folding, indentation, and structural navigation in editors like Neovim, Zed, Helix, and on GitHub for code search and navigation.

Tree-sitter targets editor developers, language tooling authors, and anyone building tools that need to understand code structure. Its grammars exist for over 200 programming languages, maintained by the community.

How it saves time or tokens

Tree-sitter parses incrementally: when you change one line of code, it re-parses only the affected portion of the syntax tree instead of re-parsing the entire file. This makes it fast enough for real-time syntax highlighting even in large files. For AI coding tools, Tree-sitter's syntax trees provide structured code understanding that is more reliable than regex-based parsing.

AI agents can use Tree-sitter to extract function signatures, class definitions, and import statements from source code without sending the entire file to an LLM, reducing token usage in code analysis tasks.

How to use

1. For editor users: Neovim, Zed, and Helix include Tree-sitter support out of the box. In Neovim, install nvim-treesitter and run :TSInstall python to add language support.
2. For developers: install the Tree-sitter CLI (npm install tree-sitter-cli) and the language grammar you need (npm install tree-sitter-python).
3. Use the library API to parse source code into a syntax tree, query it with S-expression patterns, and traverse or modify nodes programmatically.

Example

const Parser = require('tree-sitter');
const Python = require('tree-sitter-python');

const parser = new Parser();
parser.setLanguage(Python);

const code = 'def greet(name):\n    return f"Hello, {name}"';
const tree = parser.parse(code);

// Query for all function definitions
const query = new Parser.Query(Python, '(function_definition name: (identifier) @name)');
const matches = query.matches(tree.rootNode);
matches.forEach(m => console.log(m.captures[0].node.text)); // 'greet'

The query uses S-expression patterns to find structural elements in the syntax tree without fragile regex matching.

Related on TokRepo

• AI tools for coding — Tools that help with code understanding and editing
• Prompt library — Prompts for AI-assisted code analysis

Common pitfalls

• Tree-sitter grammars are language-specific and maintained separately. Some grammars may lag behind the latest language features. Check the grammar repository for your language before relying on new syntax.
• The query language uses S-expressions, which have a learning curve if you are unfamiliar with Lisp-like syntax. The Tree-sitter playground on the website helps with interactive query development.
• Tree-sitter produces concrete syntax trees (CST), not abstract syntax trees (AST). CSTs include all tokens including whitespace and punctuation, which adds verbosity but preserves perfect source fidelity.