Evaluate retrieval as a chain of evidence

Retrieval evaluation estimates how well a ranked system serves information needs. A credible evaluation connects queries, context, graded judgments, corpus snapshot, stage outputs, metrics, and qualitative error analysis. A number without those elements is not a reusable claim. BEIR demonstrates the importance of heterogeneous datasets for zero-shot retrieval; MTEB broadens embedding evaluation across tasks. Product decisions still require in-domain judgments and operational metrics.

Build an evaluation record per query. Include query ID and text or protected reference, intent class, user or policy context, time, required filters, relevant search-unit identities with grades, annotator provenance, ambiguity notes, and corpus generation. Identity-based labels survive wording changes better than copied expected snippets. Mark unjudged candidates separately from judged irrelevant ones; incomplete pools can bias systems that retrieve novel documents.

Recall at k is the fraction of relevant items retrieved in the first k. Precision at k is the fraction of the first k that is relevant. Reciprocal rank is the inverse position of the first relevant item. Average precision rewards ranking all relevant items early. Discounted cumulative gain sums graded gains discounted by position; nDCG divides by the best possible ordering for that query. Define gain mapping and treatment of no-relevance queries explicitly.

Metrics answer different products. A known-item lookup may prioritize success at one and MRR. Research discovery may value recall and facet coverage at larger k. A RAG candidate stage may prioritize evidence recall before a small context budget. Safety investigations may require all critical records, making a low-frequency slice more important than mean nDCG. State which metric is primary before comparing systems.

Evaluate stages. Corpus coverage asks whether relevant units existed and were eligible. Candidate recall measures lexical, dense, and union lists before reranking. Fusion and reranking report rank deltas. Final-set evaluation includes diversity and duplicates. Systems metrics include latency distributions, timeouts, partial shards, index lag, storage, cost, and policy correctness. A final nDCG drop becomes actionable when the relevant item can be located at a specific boundary.

Use paired comparisons because systems answer the same queries. Report per-query metric differences, bootstrap confidence intervals, and slice results. A small average gain can hide severe regression on exact identifiers. Useful slices include intent, language, source type, query length, rare entity, paraphrase, negation, time constraint, filter selectivity, popularity, and newly indexed content. Predeclare critical slices to resist selectively reporting wins.

Judgment quality matters. Give annotators the information need and enough source context, not only isolated chunks. Use multiple annotators for ambiguous high-consequence cases, record disagreement, and adjudicate with a documented rubric. Search pools from multiple systems so judgments are not biased to one baseline. Refresh judgments as the corpus changes, but preserve historical snapshot evaluations for reproducibility.

Offline relevance does not equal user success. Online experiments may measure reformulation, click or open behavior, successful downstream task, time to evidence, abandonment, and explicit usefulness. Position and presentation bias make clicks imperfect labels. Guardrail metrics should cover latency, unauthorized result count, freshness, and source diversity. Canary exposure follows offline gates; it does not replace them.

Error analysis samples losses by consequence and pattern. Label corpus missing, parse failure, filter error, analyzer mismatch, embedding miss, ANN miss, fusion loss, reranker error, duplication, stale source, or judgment defect. Quantify categories, select representative traces, and propose an intervention at the responsible layer. Re-run the fixed set after changes. Maintain a regression suite of every critical resolved failure.

The code below computes recall, reciprocal rank, and nDCG from stable identities and graded labels. Production evaluation should also validate that returned revision and policy context match the fixture. Store metric implementation version and raw per-query outputs; aggregate dashboards should be reproducible from them.

Evaluation is an operating loop: define, label, compare, inspect, intervene, and monitor. Benchmark results can guide model selection, but only a versioned product dataset reveals whether the complete system satisfies its relevance contract.