Leiden communities and hierarchical reports

Community detection compresses a large graph into groups whose internal connections are comparatively strong. In GraphRAG, communities organize distributed evidence so a global query can reason over summaries rather than raw source units. The compression is useful but lossy. A community is an algorithmic artifact produced by a graph projection, weight definition, resolution parameter, random seed, and release—not an objective natural category.

The Leiden algorithm improves on Louvain by refining partitions and addressing poorly connected communities. It optimizes a quality function through local movement, refinement, and aggregation. Its guarantees concern the partition relative to the chosen objective; they do not guarantee that the result matches domain meaning. Directed, typed, temporal, and negative relations may require projection choices because a simple weighted undirected graph discards much of their semantics.

Before clustering, define which nodes and edges participate. Including generic related-to edges, document nodes, or ubiquitous entities can dominate the partition. Weight by distinct supporting documents rather than raw mention count when repeated templates would otherwise inflate relations. Consider whether negated, disputed, or expired claims should contribute. Record every transformation from evidence graph to clustering graph.

Resolution controls granularity. A coarse partition creates broad communities and short reports but may mix topics. A fine partition yields focused groups and more reports, increasing indexing and query cost. Hierarchical clustering lets fine communities roll into coarser parents. Evaluate several levels against global query coverage and local coherence; do not select a resolution merely because the visualization looks balanced.

Community reports are generated index artifacts. A report should identify its community and level, member entities, salient relationships and claims, source-unit support, model and prompt, token budget, and release. Structure the report into typed fields such as title, synopsis, findings, and evidence references. Require each finding to cite claims or units. A fluent paragraph with no support map is unsuitable for high-confidence synthesis.

Summaries can compound errors. A false extracted edge can move nodes into a community; the report can then narrate the accidental grouping as a theme; a global answer can treat the report as evidence. Mitigate this cascade with graph anomaly checks, evidence-aware report generation, verifier passes against source units, and evaluation at each layer. Do not summarize a summary into a new source during re-indexing.

Incremental change is difficult because a few edges can alter partitions and report identities. Stable community identifiers cannot safely be based only on numeric cluster labels. Match new communities to prior ones through member overlap and lineage, assigning split, merge, new, and retired events. Rebuild affected parent reports and invalidate answer caches. For high-churn corpora, scheduled snapshots may be more understandable than continuous silent reclustering.

Security boundaries must precede clustering. A cross-tenant community report leaks structure even if restricted entity names are later removed. Construct partitions within compatible authorization domains, or prove a privacy-preserving aggregation mechanism for the specific use. Report identifiers and sizes can themselves reveal sensitive changes.

Evaluate structural and semantic quality. Structural measures include partition connectivity, stability, size distribution, and conductance-like diagnostics. Semantic review asks whether members and report findings form a coherent answerable topic. Retrieval evaluation asks whether reports improve global-answer coverage, diversity, support, and cost over passage baselines. Include adversarial cases: high-degree noise, one poisoned bridge, duplicated templates, and disconnected subgroups.

Community reports are best viewed as navigational briefs. They help select parts of the corpus and frame map-reduce work, but consequential claims should be verified against lower-level evidence.