Cette page est affichée en anglais. Une traduction française est en cours.
KnowledgeMay 19, 2026·2 min de lecture

Embedding Drift Monitoring — Retrieval Regression Runbook

Embedding drift monitoring runbook for RAG and agent search. Uses golden queries, recall@K, rank delta, and rollback gates.

henuwangkai
henuwangkai · Community
Prêt pour agents

Installation agent prête

Cet actif peut être installé après choix du runtime, vérification du plan et exécution de la commande adaptée.

Native · 98/100Policy : autoriser
Surface agent
Tout agent MCP/CLI
Type
Knowledge
Installation
Single
Confiance
Éditeur vérifié
Point d'entrée
README.md
Commande d'installation directe
npx -y tokrepo@latest install ea696ee5-0736-48e3-a789-f5a026223bd0 --target codex

À exécuter après confirmation du plan en dry-run.

Metrics That Matter

Metric Use
Recall@K on golden queries Catches lost must-return documents.
Rank delta for critical docs Shows whether important docs fell below the fold.
Top-K overlap Detects broad distribution shifts.
Empty-result rate Finds tokenizer, filter, or metadata regressions.
Click or install follow-through Confirms search quality after launch.

Vector distance alone is not enough. A lower average distance can still be worse if the wrong assets now rank above the exact answer.

Change Types To Test

  • Embedding model upgrade or provider switch.
  • Chunk size, overlap, or markdown parsing change.
  • Metadata filter changes such as visibility, asset_kind, or language.
  • Hybrid ranking weight changes between BM25 and vector score.
  • Corpus refresh that adds many near-duplicate documents.

Ship Gate

Ship only when:

  1. Must-include recall does not regress.
  2. Empty-result rate does not increase for high-intent queries.
  3. Top critical docs remain in top 3 or top 5 where expected.
  4. Any intentional ranking shift is documented with examples.
  5. Rollback is available: old index, old embedding model, or old ranker config.
🙏

Source et remerciements

This is an original TokRepo runbook by William Wang. It uses standard IR evaluation ideas such as recall@K and rank movement, and applies them to vector search systems commonly used with RAG and agent registries.

Fil de discussion

Connectez-vous pour rejoindre la discussion.
Aucun commentaire pour l'instant. Soyez le premier à partager votre avis.

Actifs similaires

MCP Latency Probe — tools/list p95 Runbook

MCP tool calling latency runbook for agents. Measures tools/list p95, separates server latency from network delay, and defines pause rules.

Knowledge
henuwangkai

LLM Prompt Caching — Cache-Key Design Runbook

LLM prompt caching techniques for agents and apps. Covers stable prefixes, cache keys, TTLs, metrics, and cached-output validation.

Knowledge
henuwangkai

Datadog LLM Observability — Trace Cost, Latency, Drift

Datadog LLM Observability traces OpenAI / Anthropic / Bedrock calls, tracks per-user cost, surfaces drift. Dashboards and span-level prompt view.

Knowledge
Datadog

Zero-Downtime DB Migration — Expand Contract Checklist

Expand-contract database migration checklist for agents. Covers additive schema changes, batched backfills, rollback, and contract gates.

Knowledge
henuwangkai
◈Accueil🔍Rechercher👤Moi