CLI ToolsMay 11, 2026·2 min read

Margin Eval — Local Evals for CLI Coding Agents

Margin Eval is an eval runtime that benchmarks CLI coding agents and records accuracy, token usage, runtime, and traces in a reproducible format.

Script Depot
Script Depot · Community
Agent ready

Safe staging for this asset

This asset is staged first. The copied prompt tells the agent to inspect the staged files and ask before activating scripts, MCP config, or global config.

Stage only · 17/100Policy: stage
Agent surface
Any MCP/CLI agent
Kind
CLI Tool
Install
Stage only
Trust
Trust: Established
Entrypoint
Asset
Safe staging command
npx -y tokrepo@latest install f4905383-abe8-46fb-8c5c-2cdcdb45b141 --target codex

Stages files first; activation requires review of the staged README and plan.

Intro

Margin Eval is an eval runtime that benchmarks CLI coding agents and records accuracy, token usage, runtime, and traces in a reproducible format.

  • Best for: Teams comparing CLI agents (Claude Code/Codex/Gemini CLI) with one unified harness and trace format
  • Works with: Docker + a provider API key or OAuth; runs local suites from Git repos and saves run bundles
  • Setup time: 20 minutes

Practical Notes

  • Setup time ~20 minutes (install + margin check + one dry-run)
  • Two measurable checks: margin --version works, and a run bundle is produced under your output folder
  • GitHub stars + forks (verified): see Source & Thanks

Margin Eval is strongest when you standardize “what counts as success” for tool-using agent runs:

  • Use a shared suite repo for scenarios and fixtures.
  • Keep agent configs in version control (so changes are reviewed).
  • Compare agents side-by-side using the same suites and eval configs.

If you run multiple providers, treat auth as part of the harness: keep keys out of logs, and make sure dry-run is part of every developer’s setup.

FAQ

Q: Why evaluate locally instead of only in CI? A: Local evals shorten iteration loops. You can reproduce a failure immediately before pushing.

Q: Do I need Docker? A: The README lists Docker as a prerequisite for the quickstart.

Q: What should I store long-term? A: Store the run bundle/traces and a small summary so regressions can be audited later.

🙏

Source & Thanks

Source: https://github.com/Margin-Lab/evals > License: AGPL-3.0 > GitHub stars: 59 · forks: 1

Discussion

Sign in to join the discussion.
No comments yet. Be the first to share your thoughts.

Related Assets

LLxprt Code — Multi-Provider AI Coding CLI

LLxprt Code is an open-source AI coding CLI that switches across providers (Anthropic, Gemini, Codex, local). Install via brew or npm.

CLI Tools
Script Depot

TokenTracker — Local Cost Dashboard for 13+ AI CLIs

TokenTracker collects token usage across 13+ AI coding tools and serves a local dashboard for cost trends, model breakdowns, and integration health.

CLI Tools
Script Depot

DSPy Micro Agent — CLI + FastAPI + Evals

evalops/dspy-micro-agent is a minimal agent runtime (CLI + FastAPI + eval harness); verified 73★ and documents `micro-agent ask` and `run_evals.py --n 50`.

CLI Tools
Agent Toolkit

Reasonix — DeepSeek-Native Coding Agent CLI

Reasonix is a DeepSeek-native coding agent CLI for the terminal; README reports a 99.82% cache hit in a real-day case study.

CLI Tools
Script Depot
◈Home🔍Search👤Me