Esta página se muestra en inglés. Una traducción al español está en curso.

KnowledgeMay 19, 2026·2 min de lectura

Embedding Drift Monitoring — Retrieval Regression Runbook

Embedding drift monitoring runbook for RAG and agent search. Uses golden queries, recall@K, rank delta, and rollback gates.

henuwangkai · Community

Listo para agents

Este activo puede ser leído e instalado directamente por agents

TokRepo expone un comando CLI universal, contrato de instalación, metadata JSON, plan según adaptador y contenido raw para que los agents evalúen compatibilidad, riesgo y próximos pasos.

Native · 98/100Política: permitir

Superficie agent

Cualquier agent MCP/CLI

Tipo

Knowledge

Instalación

Single

Confianza

Publisher verificado

Entrada

README.md

Comando CLI universal

npx tokrepo install ea696ee5-0736-48e3-a789-f5a026223bd0

contrato de instalación JSON de metadata plan adaptador contenido raw

Metrics That Matter

Metric	Use
Recall@K on golden queries	Catches lost must-return documents.
Rank delta for critical docs	Shows whether important docs fell below the fold.
Top-K overlap	Detects broad distribution shifts.
Empty-result rate	Finds tokenizer, filter, or metadata regressions.
Click or install follow-through	Confirms search quality after launch.

Vector distance alone is not enough. A lower average distance can still be worse if the wrong assets now rank above the exact answer.

Change Types To Test

Embedding model upgrade or provider switch.
Chunk size, overlap, or markdown parsing change.
Metadata filter changes such as visibility, asset_kind, or language.
Hybrid ranking weight changes between BM25 and vector score.
Corpus refresh that adds many near-duplicate documents.

Ship Gate

Ship only when:

Must-include recall does not regress.
Empty-result rate does not increase for high-intent queries.
Top critical docs remain in top 3 or top 5 where expected.
Any intentional ranking shift is documented with examples.
Rollback is available: old index, old embedding model, or old ranker config.

🙏

Fuente y agradecimientos

This is an original TokRepo runbook by William Wang. It uses standard IR evaluation ideas such as recall@K and rank movement, and applies them to vector search systems commonly used with RAG and agent registries.

Discusión

Inicia sesión para unirte a la discusión.

Aún no hay comentarios. Sé el primero en compartir tus ideas.

Activos relacionados

MCP Latency Probe — tools/list p95 Runbook

MCP tool calling latency runbook for agents. Measures tools/list p95, separates server latency from network delay, and defines pause rules.

LLM Prompt Caching — Cache-Key Design Runbook

LLM prompt caching techniques for agents and apps. Covers stable prefixes, cache keys, TTLs, metrics, and cached-output validation.

Datadog LLM Observability — Trace Cost, Latency, Drift

Datadog LLM Observability traces OpenAI / Anthropic / Bedrock calls, tracks per-user cost, surfaces drift. Dashboards and span-level prompt view.

Zero-Downtime DB Migration — Expand Contract Checklist

Expand-contract database migration checklist for agents. Covers additive schema changes, batched backfills, rollback, and contract gates.

◈Inicio 🔍Buscar 👤Yo