Agent Evaluation — Test Virtual Agents in CI

Practical Notes

• Setup time ~20 minutes (venv + install + run the smallest doc example)
• Two measurable checks: 1+ evaluated conversations, and a stable score you can compare across commits
• GitHub stars + forks (verified): see Source & Thanks

The core idea is simple: treat “agent quality” as a test suite, not a demo. Use Agent Evaluation to run the same scenarios repeatedly and record outcomes (scores, traces, and failures) so you can gate releases.

Recommended rollout:

1. Start with 5–10 deterministic scenarios (tool calls, refusals, RAG correctness).
2. Pin an evaluator config and dataset so results are comparable.
3. Run evals on every PR, and alert on score regressions or new failure modes.

If you maintain your own agent, prioritize hooks that validate tool behavior (e.g., “no destructive calls”, “no secrets in logs”) before you optimize the model.

FAQ

Q: Does this require AWS? A: No. The repo includes AWS-focused integrations, but the evaluation pattern (scenarios + scoring + CI gating) applies to any callable agent.

Q: How many scenarios should I start with? A: Start small: 5–10 scenarios that represent your highest-risk workflows, then grow coverage weekly.

Q: What should I measure? A: At minimum: pass/fail rate, stable scores, token usage, and runtime. Add safety checks for tools and data leakage.