# AI Agent Evals — GitHub Action for CI Scoring

> Microsoft AI Agent Evals is a GitHub Action that runs offline agent evaluations in CI/CD using datasets and evaluators so PRs ship with measurable gates.

## Install

Save the content below to `.claude/skills/` or append to your `CLAUDE.md`:

## Quick Use

1. Install / run:
   ```bash
   mkdir -p .github/workflows && ${EDITOR:-nano} .github/workflows/ai-agent-eval.yml
   ```
2. Start / smoke test:
   ```bash
   gh workflow run "AI Agent Evaluation" || true
   ```
3. Verify:
   - Add the sample workflow using `microsoft/ai-agent-evals@v3-beta` and confirm the Action job produces a run summary with evaluation outputs.

## Intro

Microsoft AI Agent Evals is a GitHub Action that runs offline agent evaluations in CI/CD using datasets and evaluators so PRs ship with measurable gates.

- **Best for:** Teams using GitHub Actions who want agent quality checks on every PR (before release)
- **Works with:** GitHub Actions; uses a dataset JSON and Foundry evaluators; integrates with Azure login in the sample workflow
- **Setup time:** 25 minutes

## Practical Notes

- Setup time ~25 minutes (wire workflow + prepare a tiny dataset JSON)
- Two measurable checks: Action run succeeds, and the summary contains evaluation outputs you can compare across runs
- GitHub stars + forks (verified): see Source & Thanks

The safest way to adopt this is to start with a “tiny dataset” and a single evaluator, then expand coverage. Keep your dataset versioned in-repo so it changes via PR review.

Practical CI pattern:

- Run on `pull_request` and `main` pushes.
- Fail the job when key metrics regress beyond an agreed threshold.
- Store the evaluation summary/report as an artifact for auditing.

Avoid judging only the final answer: include tool traces and system-level checks when your agents can execute actions.

### FAQ

**Q: Do I need a large dataset?**
A: No. Start with a tiny dataset, then add queries as your agent grows.

**Q: Where do results show up?**
A: The README states outputs appear in the GitHub Actions run summary section.

**Q: How do I make it a gate?**
A: Fail the job on regressions and require the check to pass before merge.

## Source & Thanks

> Source: https://github.com/microsoft/ai-agent-evals
> License: MIT
> GitHub stars: 77 · forks: 16

---

<!-- ZH -->

## 快速使用

1. 安装 / 运行：
   ```bash
   mkdir -p .github/workflows && ${EDITOR:-nano} .github/workflows/ai-agent-eval.yml
   ```
2. 启动 / 冒烟测试：
   ```bash
   gh workflow run "AI Agent Evaluation" || true
   ```
3. 验证：
   - 按 README 的 Sample workflow 引入 `microsoft/ai-agent-evals@v3-beta`；确认 Action 运行后在 Summary 区域输出评测结果。

## 简介

Microsoft AI Agent Evals 是一个 GitHub Action：在 CI/CD 中基于数据集与评估器离线跑 agent 评测，让每次 PR 都有可量化的质量闸门与可审计输出。

- **适合谁：** 使用 GitHub Actions 的团队，希望把 agent 质量检查前置到每次 PR
- **可搭配：** GitHub Actions；基于数据集 JSON 与 Foundry 评估器；示例工作流包含 Azure 登录步骤
- **准备时间：** 25 分钟

## 实战建议

- 准备时间约 25 分钟（接入工作流 + 准备一个最小数据集 JSON）
- 两项可量化检查：Action 能跑通；Summary 里有可对比的评测输出
- GitHub stars / forks（已核验）：见「来源与感谢」

最稳妥的接入方式：先用 “tiny dataset” + 1 个评估器跑通，再逐步扩展覆盖。数据集建议放在仓库内版本化，让它也走 PR 审核。

实用 CI 模式：

- 在 `pull_request` 与 `main` push 上运行。
- 关键指标回退超过阈值就让 Job 失败。
- 把评测报告/summary 保存为 artifact，便于审计与回溯。

不要只 judge 最终答案：当你的 agent 具备执行能力时，应把工具轨迹与系统级检查纳入评测。

### FAQ

**一定要大数据集吗？**
答：不需要。先用最小数据集跑通，再随着 agent 能力增长逐步补齐。

**结果显示在哪里？**
答：README 说明评测输出会出现在 GitHub Actions 的 run summary 区域。

**怎么把它变成闸门？**
答：把回归条件写成失败规则，并把该检查设为合并前必过。

## 来源与感谢

> Source: https://github.com/microsoft/ai-agent-evals
> License: MIT
> GitHub stars: 77 · forks: 16


---
Source: https://tokrepo.com/en/workflows/ai-agent-evals-github-action-for-ci-scoring
Author: Script Depot