Global search and map-reduce synthesis

Global search addresses questions whose answer is distributed across a corpus rather than centered on one entity. Examples include recurring operational risks, dominant stakeholder concerns, and differences among program areas. Retrieving the top few passages by semantic similarity can overrepresent common wording and miss diverse evidence. GraphRAG uses hierarchical community reports as a corpus map and applies a map-reduce procedure to synthesize across relevant regions.

The map stage sends the query and one or more authorized community reports to a model. Each mapper returns a typed partial result: candidate findings, support references, relevance, uncertainty, and optionally a no-evidence decision. Mappers should not see hidden communities or unrestricted reports. Their output schema must reject unreferenced findings and cap result count so verbosity does not become ranking weight.

The reduce stage combines partials under a second budget. It clusters duplicates, preserves disagreements, weighs evidence quality and diversity, and produces a supported response. Reduction is not a vote among fluent summaries. Several reports may repeat one upstream source, so count distinct evidence and source domains. A minority finding with strong direct evidence may be more important than a repeated generated phrase.

Hierarchy provides a cost-quality control. Coarse reports cover more of the corpus cheaply but blur detail. Fine reports preserve specifics but increase calls and redundancy. One plan starts at a coarse level, scores relevance and uncertainty, then descends into selected child communities. Another maps a fixed level chosen from offline evaluation. Record which reports were skipped; an answer’s coverage claim is meaningful only relative to examined scope.

Context window and call budgets can bias results. Processing the first communities until tokens run out privileges arbitrary ordering. Allocate budgets across partitions, rank within security scope, and reserve capacity for diversity. Parallel mapping lowers latency but requires deterministic identifiers and stable reduction inputs. Retries should not double-count a partial result.

Community reports are orientation, not final proof. For high-consequence findings, dereference report citations to claims and source units, retrieve representative primary spans, and verify that the synthesis does not strengthen modality or erase exceptions. This drill-down can occur during reduction or as a post-generation support pass. The answer should cite original units wherever practical and identify when a statement rests only on a generated report.

Contradictions are first-class results. If communities report opposing trends, preserve both with scope, time, and evidence rather than forcing one average story. Global queries often hide sampling differences: business units may use different reporting templates or have unequal source coverage. A synthesis should distinguish absence of a theme from lack of indexed evidence.

Evaluation needs reference sets that reward coverage and support, not exact wording. Label expected themes, critical minority findings, forbidden claims, and evidence groups. Measure theme recall, evidence precision, source diversity, contradiction preservation, citation correctness, and abstention. Compare against stratified lexical and dense retrieval, random or metadata-based sampling, and a no-graph hierarchical summarizer. Evaluate by corpus size and query class.

Global search is especially exposed to poisoning. A repeated planted narrative can influence many extracted edges, reports, and mappers. Deduplicate near-identical sources, cap contribution by source domain, flag coordinated novelty, and verify influential findings against trusted primary material. Prompt injection inside a community report is still untrusted content and cannot change reducer policy.

The result should include a coverage envelope: corpus and index release, authorization scope, community level, reports considered, reports mapped, source units verified, time window, and known gaps. Global synthesis becomes trustworthy when it is explicit about what “across the corpus” actually meant.