SkillsMay 11, 2026·2 min read

AI Agent Evals — GitHub Action for CI Scoring

Microsoft AI Agent Evals is a GitHub Action that runs offline agent evaluations in CI/CD using datasets and evaluators so PRs ship with measurable gates.

Script Depot
Script Depot · Community
Agent ready

Ready-to-run agent install

This asset can be installed after the agent chooses its runtime, checks the plan, and runs the matching command.

Native · 98/100Policy: allow
Agent surface
Any MCP/CLI agent
Kind
Skill
Install
Single
Trust
Trust: Established
Entrypoint
Asset
Direct install command
npx -y tokrepo@latest install 447b7a3c-f442-441c-846a-4f3fad62ff51 --target codex

Run after dry-run confirms the install plan.

Intro

Microsoft AI Agent Evals is a GitHub Action that runs offline agent evaluations in CI/CD using datasets and evaluators so PRs ship with measurable gates.

  • Best for: Teams using GitHub Actions who want agent quality checks on every PR (before release)
  • Works with: GitHub Actions; uses a dataset JSON and Foundry evaluators; integrates with Azure login in the sample workflow
  • Setup time: 25 minutes

Practical Notes

  • Setup time ~25 minutes (wire workflow + prepare a tiny dataset JSON)
  • Two measurable checks: Action run succeeds, and the summary contains evaluation outputs you can compare across runs
  • GitHub stars + forks (verified): see Source & Thanks

The safest way to adopt this is to start with a “tiny dataset” and a single evaluator, then expand coverage. Keep your dataset versioned in-repo so it changes via PR review.

Practical CI pattern:

  • Run on pull_request and main pushes.
  • Fail the job when key metrics regress beyond an agreed threshold.
  • Store the evaluation summary/report as an artifact for auditing.

Avoid judging only the final answer: include tool traces and system-level checks when your agents can execute actions.

FAQ

Q: Do I need a large dataset? A: No. Start with a tiny dataset, then add queries as your agent grows.

Q: Where do results show up? A: The README states outputs appear in the GitHub Actions run summary section.

Q: How do I make it a gate? A: Fail the job on regressions and require the check to pass before merge.

🙏

Source & Thanks

Source: https://github.com/microsoft/ai-agent-evals > License: MIT > GitHub stars: 77 · forks: 16

Discussion

Sign in to join the discussion.
No comments yet. Be the first to share your thoughts.

Related Assets

PR-Agent — AI PR Reviews via GitHub Action

PR-Agent is an open-source PR review agent you run as a GitHub Action or CLI to summarize changes and post review suggestions using an LLM key you control.

Skills
Agent Toolkit

promptfoo-action — Run Prompt Evals in GitHub CI

Add promptfoo-action to GitHub Actions to run prompt/agent evals on PRs or pushes, cache results, and comment a before/after report for safer iteration.

PromptsCLI Tools
Script Depot

actionlint — Lint GitHub Actions Locally

actionlint catches syntax mistakes and expression/type errors in GitHub Actions workflows before CI runs, so broken YAML never blocks your team.

SkillsCLI Tools
Script Depot

SWE-agent — Autonomous GitHub Issue Solver

SWE-agent lets LLMs autonomously fix GitHub issues and find security vulnerabilities. 18.9K+ stars. State-of-the-art on SWE-bench. MIT.

Skills
Skill Factory
◈Home🔍Search👤Me