# promptfoo-action — Run Prompt Evals in GitHub CI

> Add promptfoo-action to GitHub Actions to run prompt/agent evals on PRs or pushes, cache results, and comment a before/after report for safer iteration.

## Install

Copy the content below into your project:

# promptfoo-action — Run Prompt Evals in GitHub CI

> Add promptfoo-action to GitHub Actions to run prompt/agent evals on PRs or pushes, cache results, and comment a before/after report for safer iteration.

## Quick Use

1. Install:
   ```bash
   git checkout -b add-promptfoo-ci
   ```
2. Run:
   ```bash
   Add `uses: promptfoo/promptfoo-action@v1` to a workflow and point it at `promptfooconfig.yaml`.
   ```
3. Verify:
   - Open a PR that touches prompts and confirm the action posts (or logs) an evaluation report.


---

## Intro

Add promptfoo-action to GitHub Actions to run prompt/agent evals on PRs or pushes, cache results, and comment a before/after report for safer iteration.

- **Best for:** teams shipping prompts/agents who want CI regressions checks and a human-reviewable report in PRs
- **Works with:** GitHub Actions, promptfoo configs (YAML/JSON), and optional caching via actions/cache (per repo docs)
- **Setup time:** 13 minutes


### Quantitative Notes

- GitHub stars + forks (verified): see Source & Thanks
- Action writes results to `output.json` (repo docs)
- Setup time ~13 minutes (workflow + one config file)


---

## Practical Notes

A minimal workflow is to run evals on PRs that touch `prompts/**` and store `output.json` as an artifact. Example snippet:

```yaml
- uses: actions/checkout@v4
- uses: promptfoo/promptfoo-action@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config: promptfooconfig.yaml
```

Start with a small test set, then expand coverage once the report format fits your review process.

**Safety note:** Treat eval configs like code: review provider keys, red-team prompts, and data files; avoid leaking secrets in logs.

### FAQ

**Q: Do I need to host anything?**
A: No. It runs in GitHub Actions and uses promptfoo under the hood.

**Q: Can I gate merges on quality?**
A: Yes. Use thresholds/fail options so CI fails when success rate drops.

**Q: How do I keep costs down?**
A: Cache results and limit concurrency; run evals only on prompt-related paths.

---

## Source & Thanks

> GitHub: https://github.com/promptfoo/promptfoo-action
> Owner avatar: https://avatars.githubusercontent.com/u/137907881?v=4
> License (SPDX): MIT
> GitHub stars (verified via `api.github.com/repos/promptfoo/promptfoo-action`): 65
> GitHub forks (verified via `api.github.com/repos/promptfoo/promptfoo-action`): 31


---

<!-- ZH -->

# promptfoo-action——在 GitHub CI 跑 Prompt 评测

> 把 promptfoo-action 接入 GitHub Actions：在 PR/Push 时跑 prompt/agent 评测，支持缓存与阈值失败，并产出可对比报告，降低迭代与上线风险。

## 快速使用

1. 安装：
   ```bash
   git checkout -b add-promptfoo-ci
   ```
2. 运行：
   ```bash
   Add `uses: promptfoo/promptfoo-action@v1` to a workflow and point it at `promptfooconfig.yaml`.
   ```
3. 验证：
   - Open a PR that touches prompts and confirm the action posts (or logs) an evaluation report.


---

## 简介

把 promptfoo-action 接入 GitHub Actions：在 PR/Push 时跑 prompt/agent 评测，支持缓存与阈值失败，并产出可对比报告，降低迭代与上线风险。

- **适合谁（Best for）:** 在交付 prompts/agents 的团队，希望在 CI 里做回归检测，并在 PR 中生成可人工审阅的报告
- **兼容工具（Works with）:** GitHub Actions、promptfoo 配置（YAML/JSON），以及 actions/cache 缓存（仓库说明）
- **安装时间（Setup time）:** 13 分钟


### 量化信息

- GitHub stars + forks（已核验）：见「来源与感谢」
- Action 会写出 `output.json` 结果文件（仓库说明）
- 配置约 13 分钟（workflow + 一个配置文件）


---

## 实战要点

最小可用做法：只对 `prompts/**` 相关 PR 触发评测，并把 `output.json` 作为产物保存。先用小规模测试集跑通，再逐步扩展覆盖面，直到报告格式适配你们的评审流程。

**安全提示：** 把评测配置当作代码来审：检查 provider key、红队用例和数据文件；避免把敏感信息打进日志。

### FAQ

**Q: 需要自建服务吗？**
A: 不需要。它直接在 GitHub Actions 里跑，底层调用 promptfoo。

**Q: 能卡住合并吗？**
A: 可以。用阈值/失败策略，让成功率低于门槛时 CI 失败。

**Q: 怎么控制成本？**
A: 启用缓存、限制并发，并把触发范围限定在 prompts 目录相关变更。

---

## 来源与感谢

> GitHub：https://github.com/promptfoo/promptfoo-action
> Owner avatar：https://avatars.githubusercontent.com/u/137907881?v=4
> 许可证（SPDX）：MIT
> GitHub stars（已通过 `api.github.com/repos/promptfoo/promptfoo-action` 核验）：65
> GitHub forks（已通过 `api.github.com/repos/promptfoo/promptfoo-action` 核验）：31


---
Source: https://tokrepo.com/en/workflows/promptfoo-action-run-prompt-evals-in-github-ci
Author: Script Depot