# Agent Evaluation — Test Virtual Agents in CI

> Agent Evaluation is a Python framework that runs repeatable, scored tests for virtual agents, so teams can catch regressions automatically in CI.

## Install

Save the content below to `.claude/skills/` or append to your `CLAUDE.md`:

## Quick Use

1. Install / run:
   ```bash
   python -m venv .venv && source .venv/bin/activate && pip install agent-evaluation
   ```
2. Start / smoke test:
   ```bash
   python -c "import agenteval; print('agenteval import ok')"
   ```
3. Verify:
   - Open the documentation site and run one minimal example end-to-end; confirm an eval report is produced for at least one multi-turn conversation.

## Intro

Agent Evaluation is a Python framework that runs repeatable, scored tests for virtual agents, so teams can catch regressions automatically in CI.

- **Best for:** Teams shipping virtual agents who want a CI-grade eval harness (multi-turn + scoring)
- **Works with:** Any target agent you can call (API/SDK); built-in support includes Amazon Bedrock, Amazon Q Business, and SageMaker
- **Setup time:** 20 minutes

## Practical Notes

- Setup time ~20 minutes (venv + install + run the smallest doc example)
- Two measurable checks: 1+ evaluated conversations, and a stable score you can compare across commits
- GitHub stars + forks (verified): see Source & Thanks

The core idea is simple: treat “agent quality” as a test suite, not a demo. Use Agent Evaluation to run the same scenarios repeatedly and record outcomes (scores, traces, and failures) so you can gate releases.

Recommended rollout:

1. Start with 5–10 deterministic scenarios (tool calls, refusals, RAG correctness).
2. Pin an evaluator config and dataset so results are comparable.
3. Run evals on every PR, and alert on score regressions or new failure modes.

If you maintain your own agent, prioritize hooks that validate tool behavior (e.g., “no destructive calls”, “no secrets in logs”) before you optimize the model.

### FAQ

**Q: Does this require AWS?**
A: No. The repo includes AWS-focused integrations, but the evaluation pattern (scenarios + scoring + CI gating) applies to any callable agent.

**Q: How many scenarios should I start with?**
A: Start small: 5–10 scenarios that represent your highest-risk workflows, then grow coverage weekly.

**Q: What should I measure?**
A: At minimum: pass/fail rate, stable scores, token usage, and runtime. Add safety checks for tools and data leakage.

## Source & Thanks

> Source: https://github.com/awslabs/agent-evaluation
> License: Apache-2.0
> GitHub stars: 360 · forks: 49

---

<!-- ZH -->

## 快速使用

1. 安装 / 运行：
   ```bash
   python -m venv .venv && source .venv/bin/activate && pip install agent-evaluation
   ```
2. 启动 / 冒烟测试：
   ```bash
   python -c "import agenteval; print('agenteval import ok')"
   ```
3. 验证：
   - 打开官方文档跑通一个最小示例；确认至少 1 组多轮对话能产出评测报告（含分数/判定）。

## 简介

Agent Evaluation 是一个 Python 评测框架：用可复现的多轮对话+评分，把虚拟 Agent 的行为测试接入 CI，持续发现回归，并把分数/轨迹沉淀为可追踪的对比报告，便于持续迭代。

- **适合谁：** 在生产环境迭代虚拟 Agent 的团队，需要“可回归、可量化”的评测框架
- **可搭配：** 可通过 API/SDK 调用的任意目标 Agent；内置示例支持 Amazon Bedrock / Amazon Q Business / SageMaker
- **准备时间：** 20 分钟

## 实战建议

- 准备时间约 20 分钟（建环境 + 安装 + 跑最小示例）
- 两项可量化检查：至少 1 组对话被评测，并且分数可用于跨提交对比
- GitHub stars / forks（已核验）：见「来源与感谢」

核心原则：把“Agent 质量”当作测试套件，而不是演示。用 Agent Evaluation 把同一批场景反复跑出可对比结果（分数/轨迹/失败原因），再把它变成发布闸门。

建议落地路径：

1. 先做 5–10 个可复现场景（工具调用、拒绝策略、RAG 事实性）。
2. 固定 evaluator 配置与数据集，确保跨提交可对比。
3. 每次 PR 都跑评测；分数回退或出现新失败模式就阻断合并。

如果你在迭代自研 Agent，先用 hooks 把工具行为守住（例如“禁止破坏性调用”“日志不泄密”），再谈模型优化。

### FAQ

**一定要用 AWS 吗？**
答：不需要。仓库包含 AWS 相关集成示例，但“场景 + 评分 + CI 闸门”的方法适用于任何可调用的 Agent。

**一开始做多少场景合适？**
答：从 5–10 个高风险工作流开始，每周逐步扩展覆盖面。

**应该量化哪些指标？**
答：至少包括：通过率/失败原因、稳定分数、token 用量与耗时；并加入工具安全与数据泄露检查。

## 来源与感谢

> Source: https://github.com/awslabs/agent-evaluation
> License: Apache-2.0
> GitHub stars: 360 · forks: 49


---
Source: https://tokrepo.com/en/workflows/agent-evaluation-test-virtual-agents-in-ci
Author: Agent Toolkit