Reading tools and contents
3 min
Production RAG & GraphRAG

Chapter 9 of 10

GraphRAG evaluation by layer and query slice

Quality engineering

About 3 minutes · includes examples, an exercise, and references

Chapter at a glance

  • •Benchmark query classes and hard slices that reflect the product’s reason for using graphs.
  • •Label evidence, entities, paths, allowed inference, and abstention—not only answer text.
  • •Use controlled ablations against credible non-graph baselines.

End-to-end answer ratings cannot diagnose a GraphRAG system. The same wrong answer can arise from missing documents, bad chunking, extraction error, entity over-merge, a poor partition, router error, retrieval pruning, context packing, unsupported generation, or stale authorization. Evaluation must preserve the pipeline’s causal structure and then test the complete user outcome.

Build a query taxonomy before a benchmark. Include local attributes, relationship and path questions, global themes, temporal change, comparison, aggregation, contradiction, and unanswerable cases. Label hop count, entity popularity, evidence dispersion, document type, language, sensitivity, and expected freshness. A random sample dominated by popular one-hop questions will hide the reason GraphRAG exists.

Maintain source-level ground truth where possible. For each question, record acceptable answers, required and alternative evidence units, relevant entities and relations, temporal scope, allowed inference, forbidden claims, and abstention expectation. Some global questions allow several valid syntheses; use a rubric for theme coverage, critical minority findings, evidence diversity, and qualification rather than exact strings.

Index evaluation covers parsing and chunk boundary integrity; entity and relation extraction; mention resolution; claim polarity and time; edge evidence; graph connectivity; community coherence; report support; embedding presence; and security-label propagation. Every metric should be sliced. A 95 percent extraction score can hide near-zero performance on tables that contain the decisions users need.

Retrieval evaluation measures seed accuracy, entity and source recall at k, path validity, community-report coverage, fused-rank quality, evidence diversity, and recall after packing. Record whether the needed evidence existed in any channel. This separates an index miss from a fusion or packing miss. Evaluate authorized views because unrestricted recall is irrelevant to a user who cannot access the evidence.

Generation evaluation separates correctness, completeness, citation precision, citation completeness, entailment strength, contradiction preservation, calibration, and abstention. Use trained reviewers for consequential slices and report agreement. Model judges can accelerate comparison but need blinded ordering, position-bias controls, calibration against human labels, and versioning.

Compare systems through controlled ablations. Remove graph edges, communities, dense retrieval, reranking, or verification one at a time while holding corpus, model, prompt, and budget constant. Compare against credible lexical and dense baselines. Report confidence intervals or paired uncertainty, not only mean point estimates. Analyze regressions even when the overall score rises.

Online evaluation observes accepted answers, citation opens, reformulations, abstention recovery, latency, cost, and reported errors, but user clicks are not truth. Use canary index releases and shadow queries to compare evidence sets without exposing experimental answers. Protect evaluation data from becoming extraction prompt examples or training leakage.

Adversarial suites should include alias collisions, poisoned bridge edges, duplicated narratives, prompt injection, stale sources, conflicting dates, permission changes, missing embeddings, broken community reports, and budget exhaustion. Chaos tests can remove one index component and verify graceful fallback or typed abstention.

Define launch gates by query slice and consequence. A low-risk exploratory tool may tolerate some unsupported breadth; a compliance answer may require near-perfect citation support and high abstention. Publish a model-and-index card describing corpus, releases, test design, limitations, excluded scopes, and owner. Evaluation is an operating system for change, not a one-time score.

Key points

  • Benchmark query classes and hard slices that reflect the product’s reason for using graphs.
  • Label evidence, entities, paths, allowed inference, and abstention—not only answer text.
  • Use controlled ablations against credible non-graph baselines.
  • Set release gates by consequence and preserve regression analysis.

A layered evaluation fixture

Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.

A layered evaluation fixtureyaml
id: impact-017
query: Which workflows depend on package P-9?
class: local-multi-hop
slice: [two-hop, rare-entity, authorization-public]
expected:
  seed_entities: [package/P-9]
  required_claims: [claim-21, claim-27]
  acceptable_answers: [workflow/W-3]
  forbidden_answers: [workflow/W-8]
  max_hops: 2
  abstain: false
gates:
  citation_precision: 1.0
  evidence_recall_at_20: 1.0

Worked examples

Toy

A misleading average

Overall evidence recall is high, but every table-derived temporal query fails.

Slice analysis identifies the parser and context reconstruction defect. Adding more graph traversal would not address the failure.

  • Query slice
  • First failing layer
  • Targeted remediation

Application

Graph ablation

GraphRAG appears better than dense retrieval on impact questions.

Hold generation and budget fixed, remove graph paths, and measure paired changes. Inspect whether gains come from true distributed evidence or simply extra context tokens.

  • Controlled variables
  • Paired queries
  • Token-budget parity

Exercise

Create a release-grade evaluation matrix

Build at least thirty fixtures across query classes, evidence dispersion, permissions, and unanswerable cases.

  1. Label pipeline-level ground truth.
  2. Run baselines and ablations.
  3. Add adversarial and component-outage cases.
  4. Define slice-specific launch and rollback gates.

Success criteria

  • Failures identify the first broken layer.
  • Comparisons hold corpus and budget constant.
  • High-consequence slices cannot hide in averages.
  • The benchmark, judge versions, and index release are reproducible.

Reflect: Which metric would improve if the system became more verbose while becoming less supported?

References and further reading