Evaluate and diagnose the complete RAG chain

RAG evaluation is difficult because a plausible answer can arise from a weak chain, and a poor answer can hide strong retrieval. Evaluate components and the end-to-end outcome with the same stable identities. RAGAS proposed reference-free dimensions including faithfulness, answer relevance, and context relevance. RAGChecker decomposes retrieval and generation behavior at claim level. These frameworks provide useful methods, but product evaluation still requires explicit rubrics, trusted fixtures, and validation of automated judges.

Start with a case schema. Record request, actor and policy context, corpus generation, required scope, reference claims where available, acceptable evidence-unit identities or source constraints, expected abstention or conflict behavior, risk class, and slices. Include both answerable and deliberately unanswerable cases. A case without the searched corpus snapshot cannot be replayed after documents change.

Measure ingestion and retrieval. Corpus coverage asks whether acceptable evidence existed and was indexed. Candidate recall reports whether it appeared before context selection. Retrieval precision estimates distraction. Rank metrics show ordering. Policy correctness requires zero unauthorized evidence. Freshness verifies the correct effective revision. These metrics use identities and metadata rather than asking a generator to guess whether two texts look related.

Measure context construction. Evidence coverage asks whether the packet contains support for each required reference claim or facet. Context precision asks how much selected evidence is useful. Report duplication, authority distribution, conflict representation, token count, and omissions by reason. A relevant unit found at rank five but dropped for redundant rank-one chunks is a context failure, not a retriever failure.

Measure generation at claim level. Segment the response into material claims. For each, label supported, partially supported, contradicted, or unsupported by cited evidence. Check citation handle validity and citation precision. Completeness measures whether required claims were addressed. Correctness compares with trusted references or expert judgments when available. Style and usefulness are separate; a clear unsupported claim should not score as grounded because it is readable.

Evaluate abstention as classification. True positives are unsafe or unanswerable cases correctly declined; false positives are answerable cases unnecessarily declined. Report precision and recall for abstention, plus clarification and escalation correctness. Consequence may justify different thresholds by task. Calibration curves can compare confidence or support score with observed correctness, but only within the model and corpus version evaluated.

Automated judges reduce cost but introduce model bias, prompt sensitivity, and correlated failure. Validate each judge on a human-labelled calibration set by slice. Record judge model, prompt, temperature, and evidence. Use deterministic checks for handles, identities, dates, schemas, and policy. Use multiple judges or human adjudication for high-consequence disagreement. Do not allow the same generated answer to smuggle instructions into a judge prompt without clear isolation.

Run controlled ablations. Compare no retrieval, lexical only, dense only, hybrid, different chunking, context selection policies, generator versions, and verification. Hold other components fixed. Repeat stochastic generation. Report paired per-case differences and confidence intervals rather than only means. Temporal splits reveal whether performance relies on near-duplicate historical questions.

Maintain an error taxonomy across the chain: source absent, parse failure, wrong unit, policy filter, stale revision, query plan, candidate miss, ANN miss, fusion loss, context omission, injection, citation fabrication, unsupported synthesis, contradiction, incomplete answer, excessive abstention, evaluator error, timeout, and budget stop. Every production incident should map to or extend this taxonomy and add a fixture.

Online evaluation connects to outcomes: task completion, time to verified answer, escalation resolution, correction rate, user-reported unsupported claim, citation open behavior, latency, and cost. Clicks are not truth and can reward confident prose. Canary a bounded population after offline gates. Sample traces by risk and anomaly, with privacy controls.

A release report should show critical invariants first, then stage metrics, final metrics, slices, uncertainty, operational cost, known limitations, and representative errors. No single “RAG score” captures the system. The evidence chain makes trade-offs visible: a model may improve fluency while reducing faithfulness, or a context policy may improve precision while missing minority-language evidence.

Evaluation is also a design tool. When each claim and unit has identity, the team can ask counterfactual questions: Would the generator succeed with oracle context? Would the retriever find oracle units without ANN approximation? Would verification catch the observed unsupported claim? Those experiments identify the next engineering investment.