LLM Eval & Guardrails

What's in this pack

This pack assembles the five open-source tools every team converges on once their LLM features ship to real users and "the model got worse this week" stops being a tolerable answer. The tools split into two halves: pre-deploy evaluation (score every prompt change) and runtime guardrails (constrain what the model actually outputs).

The split matters. Pre-deploy eval catches the regression before customers see it. Runtime guardrails catch the regression you didn't predict. You need both — eval alone misses adversarial inputs you didn't sample, guardrails alone don't tell you which prompt change caused the drift.

Why eval is now table-stakes

Three forcing functions made eval the difference between teams that ship and teams that stall:

• Model upgrades happen on the vendor's schedule. When Anthropic releases Sonnet 4.7, your prompt that worked on 4.6 may behave subtly differently. Without an eval suite, you discover this from a customer support ticket. With Promptfoo, you run promptfoo eval -c promptfooconfig.yaml --providers anthropic:claude-4.7,anthropic:claude-4.6 and see the diff in 30 seconds.
• Prompts have no compile errors. A typo in code throws an exception. A typo in a prompt produces plausible but worse output that ships. Eval is the compile step prompts never had.
• RAG quality decays silently. A new doc that gets retrieved but isn't actually relevant lowers answer quality without raising any error. Ragas gives you context precision and faithfulness scores per query, so you spot decay before it accumulates.

Install in one command

# Install the entire pack into the current project
tokrepo install pack/llm-eval-guardrails

# Or pick individual assets
tokrepo install promptfoo
tokrepo install ragas

The TokRepo CLI sets up a evals/ directory with example test cases, a promptfooconfig.yaml, a Ragas notebook seeded with your retriever, and a Guardrails AI rail file template. CI snippets gate merges on eval-suite pass rate.

Common pitfalls

• LLM-as-judge without grounding. DeepEval and Ragas use a judge model to score answers, but if the judge is the same model as the system under test, you get optimistic scores. Use a different model family as judge, or pin a stronger model (e.g. judge with Claude when scoring GPT outputs).
• Eval suites with 5 cases. Five hand-picked examples don't cover the long tail. Aim for 50-200 cases derived from real production logs (Opik makes this easy — sample bad outputs, label them, promote to eval set).
• Treating Guardrails as a magic filter. Guardrails enforces structure (valid JSON, profanity-free, schema-conformant) — it doesn't catch factually wrong but well-formatted answers. Pair it with a Ragas faithfulness check.
• Running eval against production traffic costs. Eval suites can hit 5-10x your normal LLM bill if you re-score every nightly. Cache embeddings, sample your eval set per run, or use cheaper models for the judge step.
• No eval for non-text outputs. If your agent emits tool calls, eval the tool call shape with structured assertions, not just the final text. Promptfoo supports this via custom transform and assert hooks.

A typical week with this stack

Monday morning the on-call engineer reviews Opik traces from the weekend, samples 30 outputs that scored low on faithfulness, and promotes the worst 10 into the DeepEval test set. Wednesday a product manager asks whether switching the customer-support agent from GPT-4o to Claude Sonnet would save money without quality loss — the team writes a Promptfoo config, runs promptfoo eval over 200 cases against both models, and answers in 20 minutes with a side-by-side table. Friday before deploy, the CI pipeline runs the full DeepEval + Ragas suite; one regression on the new prompt blocks merge until fixed.

Throughout, Guardrails sits inline in production, rejecting outputs that fail JSON schema validation and surfacing reask rate to a Grafana dashboard. When reask rate spikes above 3%, an alert fires and the team knows the upstream prompt drifted before any customer notices.

When this pack alone isn't enough

For full production observability beyond Opik (latency percentiles, cost tracking per user, model-routing analytics), look at LangSmith or Arize Phoenix — neither is in the pack because they're more orchestration than eval. For safety classifiers (jailbreak detection, prompt injection scoring), add Llama Guard or NVIDIA NeMo Guardrails — Guardrails AI focuses on output validation, not adversarial input detection. And if your eval needs human-in-the-loop annotation at scale, Argilla or Label Studio plug into the Opik dataset format.