CLI ToolsMay 14, 2026·2 min read

tokenu — du-like Token Counter for Repos

tokenu is a du-like CLI to measure token usage per file/dir for LLM context planning; verified 59★ with `npx tokenu .` and JSON output for agents.

Script Depot
Script Depot · Community
Intro

tokenu is a du-like CLI to measure token usage per file/dir for LLM context planning; verified 59★ with npx tokenu . and JSON output for agents.

Best for: Agent builders who need to budget context windows and avoid accidentally loading 40K-token folders

Works with: Node.js + npx/npm, agent pipelines that can read JSON output, hook-based pre-read guards

Setup time: 2-6 minutes

Key facts (verified)

  • GitHub: 59 stars · 1 forks · pushed 2026-05-13.
  • License: Apache-2.0 · owner avatar + repo URL verified via GitHub API.
  • README-backed entrypoint: npx tokenu ..

Main

  • Use it before feeding a repo into an agent: token counts show which folders (like dist/) will blow up your context budget.

  • Prefer --json for agent pipelines so the agent can plan what to read within a fixed token limit.

  • If token counts vary between models, use --model / --encoding to compare costs across tokenizers.

Source-backed notes

  • README shows a no-install quick start: npx tokenu ..
  • README documents JSON output for agent consumption: tokenu --json.
  • README lists encodings (o200k_base, cl100k_base, etc.) and flags like --max-depth and --exclude.

FAQ

  • Does tokenu reduce token usage automatically?: No — README says it measures token cost so you can decide what to exclude or summarize.
  • Can an agent use tokenu programmatically?: Yes — use tokenu --json so the agent can parse per-dir token counts.
  • Is it an estimate or real tokenization?: README says it uses actual tokenization via a tokenizer library, not heuristics.
🙏

Source & Thanks

Source: https://github.com/lirantal/tokenu > License: Apache-2.0 > GitHub stars: 59 · forks: 1

Discussion

Sign in to join the discussion.
No comments yet. Be the first to share your thoughts.

Related Assets

snip — Token-Saving CLI Proxy for AI Tools

snip is a token-saving CLI proxy: declarative YAML filters cut 60–90%+ LLM context from noisy command output. Verified 232★; setup ~5–10 minutes.

CLI Tools
Script Depot

oidc-agent — OIDC Tokens for Scripts (ssh-agent)

oidc-agent manages OpenID Connect tokens like ssh-agent: start the agent, create configs with oidc-gen, then fetch tokens for scripts via oidc-token.

CLI Tools
Script Depot

GritQL — Declarative Code Rewrite CLI

GritQL is a declarative language + CLI for searching and rewriting codebases with snippet-like patterns, making large refactors repeatable and reviewable.

CLI Tools
Script Depot

OpenAnt — Verified Vuln Pipeline CLI (Go + Python)

OpenAnt is a defensive vulnerability discovery CLI: it parses a repo, analyzes findings, and runs verification steps so security output is evidence-backed.

CLI Tools
Script Depot
◈Home🔍Search👤Me