KnowledgeMay 19, 2026·2 min read

Embedding Drift Monitoring — Retrieval Regression Runbook

Embedding drift monitoring runbook for RAG and agent search. Uses golden queries, recall@K, rank delta, and rollback gates.

henuwangkai
henuwangkai · Community
Agent ready

This asset can be read and installed directly by agents

TokRepo exposes a universal CLI command, install contract, metadata JSON, adapter-aware plan, and raw content links so agents can judge fit, risk, and next actions.

Native · 98/100Policy: allow
Agent surface
Any MCP/CLI agent
Kind
Knowledge
Install
Single
Trust
Verified publisher
Entrypoint
README.md
Universal CLI install command
npx tokrepo install ea696ee5-0736-48e3-a789-f5a026223bd0
install contractmetadata JSONadapter planraw content

Metrics That Matter

Metric Use
Recall@K on golden queries Catches lost must-return documents.
Rank delta for critical docs Shows whether important docs fell below the fold.
Top-K overlap Detects broad distribution shifts.
Empty-result rate Finds tokenizer, filter, or metadata regressions.
Click or install follow-through Confirms search quality after launch.

Vector distance alone is not enough. A lower average distance can still be worse if the wrong assets now rank above the exact answer.

Change Types To Test

  • Embedding model upgrade or provider switch.
  • Chunk size, overlap, or markdown parsing change.
  • Metadata filter changes such as visibility, asset_kind, or language.
  • Hybrid ranking weight changes between BM25 and vector score.
  • Corpus refresh that adds many near-duplicate documents.

Ship Gate

Ship only when:

  1. Must-include recall does not regress.
  2. Empty-result rate does not increase for high-intent queries.
  3. Top critical docs remain in top 3 or top 5 where expected.
  4. Any intentional ranking shift is documented with examples.
  5. Rollback is available: old index, old embedding model, or old ranker config.
🙏

Source & Thanks

This is an original TokRepo runbook by William Wang. It uses standard IR evaluation ideas such as recall@K and rank movement, and applies them to vector search systems commonly used with RAG and agent registries.

Discussion

Sign in to join the discussion.
No comments yet. Be the first to share your thoughts.

Related Assets

MCP Latency Probe — tools/list p95 Runbook

MCP tool calling latency runbook for agents. Measures tools/list p95, separates server latency from network delay, and defines pause rules.

Knowledge
henuwangkai

LLM Prompt Caching — Cache-Key Design Runbook

LLM prompt caching techniques for agents and apps. Covers stable prefixes, cache keys, TTLs, metrics, and cached-output validation.

Knowledge
henuwangkai

Datadog LLM Observability — Trace Cost, Latency, Drift

Datadog LLM Observability traces OpenAI / Anthropic / Bedrock calls, tracks per-user cost, surfaces drift. Dashboards and span-level prompt view.

Knowledge
Datadog

Zero-Downtime DB Migration — Expand Contract Checklist

Expand-contract database migration checklist for agents. Covers additive schema changes, batched backfills, rollback, and contract gates.

Knowledge
henuwangkai
◈Home🔍Search👤Me