Build an evidence corpus with lineage

RAG begins before retrieval. The evidence corpus determines what the system can support, which versions it can distinguish, and whether a citation resolves to an authoritative source. Ingestion converts source objects into evidence units while preserving lineage, structure, policy, time, and quality. If this transformation is lossy or unaudited, later grounding claims are weak regardless of model performance.

Define authority by use case. A source registry records owner, source system, trust class, allowed purposes, jurisdictions, update mechanism, retention, and escalation contact. Authority is not a single global rank. A product manual may be authoritative for configuration, an incident ticket for what happened, and a policy database for permitted action. Retrieval scores relevance; source policy governs whether evidence may support a particular claim.

Use stable identities at three levels: source, immutable revision, and evidence unit. An evidence unit may be a section, paragraph group, table row with headers, code block with signature, transcript segment, or image plus verified description. It includes a locator back to the source and parent structure. A citation should resolve to the exact revision searched, not whatever content currently lives at a mutable URL.

Parse structure instead of flattening files. Headings establish scope; lists share qualifiers; tables require column headers; code needs language and surrounding explanation; PDFs may reorder columns; slides use visual grouping; tickets include chronology and participants. Test parsers with format-specific gold fixtures. Store raw source and normalized text where policy allows, plus parser version and warnings. Quarantine unreadable or ambiguous objects rather than silently indexing empty text.

Chunking should optimize evidence use, not arbitrary token counts. Begin at semantic boundaries, split oversized units, and add limited overlap only when needed. Preserve parent and adjacency links so context construction can expand around a precise hit. An evidence unit should be independently interpretable enough for ranking and citation. If a rule starts with “except when” because its condition was left in the previous chunk, the segmentation failed.

Chunking interacts with generation. Very small units increase recall opportunities but may omit definitions and create repetitive context. Large units retain coherence but consume the context budget and may contain conflicting topics. Evaluate retrieval recall, context precision, citation granularity, support, duplication, and cost across treatments. Do not select chunk size solely from an embedding model’s maximum input.

Temporal metadata must express business validity. Ingestion time, source modification time, publication time, and effective interval answer different questions. An old revision may be correct for a historical query but unsafe for current instructions. Preserve supersession relations. At query time, scope evidence to the requested or current interval and tell the generator which time applies. Conflicting revisions should not be merged into one anonymous passage.

Access control travels with every derived object. Attach tenant, classification, groups, purpose restrictions, and residency as structured policy fields rather than prose. Propagate changes to lexical indexes, vector stores, caches, summaries, and evaluation artifacts. Avoid embedding secret access labels into vectors. Authorization is checked during retrieval and before returning context to the generator, whose provider and region must also be permitted to process it.

Build a generation manifest containing source snapshot or watermarks, successful and failed objects, parser and segmentation versions, embedding contract, schema, unit counts, language distribution, duplicate statistics, and build provenance. Validate deterministic probes and sampled renderings before atomic promotion. Partial ingestion should produce an explicit coverage state; silent gaps encourage confident answers from incomplete evidence.

Deduplication needs provenance. Exact content hashes identify byte- or normalization-level copies. Near-duplicate detection can group templates or mirrors, but do not discard distinct authority, version, or policy. Keep canonical groups and lineage. Context construction can select one representative while citations still identify the correct source.

Deletion and correction are first-class. A source owner must be able to revoke an object and find every derived unit, vector, cache entry, summary, judgment, and answer artifact governed by policy. Tombstones prevent offline replicas from resurrecting content. Test deletion propagation and urgent security revocation. The corpus is not a one-time embedding job; it is a governed, continuously reconciled evidence store.