Cette page est affichée en anglais. Une traduction française est en cours.
KnowledgeMay 19, 2026·2 min de lecture

Embedding Drift Monitoring — Retrieval Regression Runbook

Embedding drift monitoring runbook for RAG and agent search. Uses golden queries, recall@K, rank delta, and rollback gates.

henuwangkai
henuwangkai · Community
Prêt pour agents

Cet actif peut être lu et installé directement par les agents

TokRepo expose une commande CLI universelle, un contrat d'installation, le metadata JSON, un plan selon l'adaptateur et le contenu raw pour aider les agents à juger l'adaptation, le risque et les prochaines actions.

Native · 98/100Policy : autoriser
Surface agent
Tout agent MCP/CLI
Type
Knowledge
Installation
Single
Confiance
Éditeur vérifié
Point d'entrée
README.md
Commande CLI universelle
npx tokrepo install ea696ee5-0736-48e3-a789-f5a026223bd0
contrat d'installationJSON metadataplan adaptateurcontenu raw

Metrics That Matter

Metric Use
Recall@K on golden queries Catches lost must-return documents.
Rank delta for critical docs Shows whether important docs fell below the fold.
Top-K overlap Detects broad distribution shifts.
Empty-result rate Finds tokenizer, filter, or metadata regressions.
Click or install follow-through Confirms search quality after launch.

Vector distance alone is not enough. A lower average distance can still be worse if the wrong assets now rank above the exact answer.

Change Types To Test

  • Embedding model upgrade or provider switch.
  • Chunk size, overlap, or markdown parsing change.
  • Metadata filter changes such as visibility, asset_kind, or language.
  • Hybrid ranking weight changes between BM25 and vector score.
  • Corpus refresh that adds many near-duplicate documents.

Ship Gate

Ship only when:

  1. Must-include recall does not regress.
  2. Empty-result rate does not increase for high-intent queries.
  3. Top critical docs remain in top 3 or top 5 where expected.
  4. Any intentional ranking shift is documented with examples.
  5. Rollback is available: old index, old embedding model, or old ranker config.
🙏

Source et remerciements

This is an original TokRepo runbook by William Wang. It uses standard IR evaluation ideas such as recall@K and rank movement, and applies them to vector search systems commonly used with RAG and agent registries.

Fil de discussion

Connectez-vous pour rejoindre la discussion.
Aucun commentaire pour l'instant. Soyez le premier à partager votre avis.

Actifs similaires

MCP Latency Probe — tools/list p95 Runbook

MCP tool calling latency runbook for agents. Measures tools/list p95, separates server latency from network delay, and defines pause rules.

Knowledge
henuwangkai

LLM Prompt Caching — Cache-Key Design Runbook

LLM prompt caching techniques for agents and apps. Covers stable prefixes, cache keys, TTLs, metrics, and cached-output validation.

Knowledge
henuwangkai

Datadog LLM Observability — Trace Cost, Latency, Drift

Datadog LLM Observability traces OpenAI / Anthropic / Bedrock calls, tracks per-user cost, surfaces drift. Dashboards and span-level prompt view.

Knowledge
Datadog

Zero-Downtime DB Migration — Expand Contract Checklist

Expand-contract database migration checklist for agents. Covers additive schema changes, batched backfills, rollback, and contract gates.

Knowledge
henuwangkai
◈Accueil🔍Rechercher👤Moi