Reading tools and contents
4 min
Retrieval Engineering

Chapter 2 of 10

Model the corpus before choosing an index

Data architecture

About 4 minutes · includes examples, an exercise, and references

Chapter at a glance

  • •Source, revision, and search-unit identities solve different lifecycle problems.
  • •Structure-aware segmentation should be evaluated, not assumed.
  • •Analyzer, parser, and embedding versions belong in an index manifest.

A retrieval engine searches a representation of a corpus, not the source system itself. The representation is produced by an ingestion pipeline that discovers content, authenticates access, parses formats, selects fields, creates searchable units, attaches policy metadata, and publishes index versions. Most persistent retrieval failures originate here. A sophisticated ranker cannot recover text that was never parsed, a table flattened into nonsense, an access label dropped, or a current policy hidden behind an obsolete duplicate.

Define three identities. A source identity names the authoritative object, such as a repository path plus commit, database primary key, or document-management identifier. A revision identity names an immutable version of that object, ideally by a content digest or source version. A search-unit identity names a passage, row, section, image description, or other addressable fragment derived from a revision. Never use an array position as durable identity. Stable identities support idempotent updates, deletions, citations, evaluation joins, and incident scoping.

Document boundaries carry meaning. Titles, headings, lists, code blocks, tables, captions, and section ancestry affect interpretation. Blind fixed-width chunking can separate a qualification from its rule or a function signature from its explanation. Structure-aware segmentation begins with source-native boundaries, then splits oversized sections and optionally adds bounded overlap. Store parent-child relations so a passage can be scored precisely and presented with enough surrounding context. Keep the canonical text separate from any embedding-specific formatting.

Chunk size is a coupled design choice. Small passages can improve localization and reduce irrelevant context, but they may omit prerequisites and create many near-duplicates. Large units retain narrative coherence but dilute lexical term frequency, make dense representations average unrelated topics, and consume reranking or generation budgets. Evaluate chunking as a treatment: hold the corpus and query set stable, then compare candidate recall, final ranking, duplication, latency, storage, and citation usefulness. Average token count alone says little.

Fields should reflect ranking and filtering needs. Common fields include title, body, heading path, author, language, tenant, access groups, effective interval, source type, tags, product version, jurisdiction, and ingestion timestamp. Decide which fields are searchable, boostable, filterable, returnable, or confidential. Do not concatenate private metadata into an embedding merely because it is convenient; vector similarity can expose correlations even if the raw field is hidden.

Normalization must be reversible enough for evidence. Unicode normalization, case folding, stemming, stopword handling, and markup removal can improve matching, but exact codes and names may be damaged. Preserve raw text and record analyzer version. Use field-specific analyzers: prose, source code, identifiers, and CJK text require different tokenization. Detect language rather than applying an English pipeline globally.

Ingestion should be idempotent and monotonic. Reprocessing the same source revision produces the same search units. Publishing a new index snapshot should not mix incompatible analyzer or embedding versions invisibly. A practical manifest records corpus snapshot, parser version, segmentation policy, model identity, tokenizer, schema, document count, failed sources, and timestamps. Build into a new generation, validate it, then atomically promote an alias. Retain enough history to reproduce evaluation and roll back.

Deletion is part of correctness. A source removal, permission change, legal hold, or retention expiry must propagate through lexical indexes, vector stores, caches, replicas, derived summaries, and evaluation fixtures according to policy. Maintain tombstones or change logs so an offline shard cannot resurrect deleted content on recovery. Test deletion latency as an SLO and verify by stable identity rather than approximate text search.

Quality checks belong before publication. Validate schema and access labels; reject impossible effective intervals; count empty or abnormally large units; sample parse output by format; compare source and indexed counts; detect duplicate content; measure language distribution; and inspect embedding norms. Quarantine failures instead of silently dropping them. Publish an index-readiness report that distinguishes complete, partial, and blocked domains.

Worked at application scale, a product-documentation corpus may have one source page per version, section passages as search units, code blocks preserved as child units, and product version as a hard filter. When a page changes, the pipeline creates a new revision, diffs derived units, upserts changed identities, deletes retired ones, embeds only changed text, validates counts, and promotes a coherent generation. Every result can then resolve to the exact source heading and revision that was searched.

Key points

  • Source, revision, and search-unit identities solve different lifecycle problems.
  • Structure-aware segmentation should be evaluated, not assumed.
  • Analyzer, parser, and embedding versions belong in an index manifest.
  • Updates, permission changes, and deletions must propagate through every derived store.

Create deterministic passage identities

Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.

Create deterministic passage identitiespython
from hashlib import sha256
from dataclasses import dataclass

@dataclass(frozen=True)
class Passage:
    source_id: str
    revision: str
    heading_path: tuple[str, ...]
    ordinal: int
    text: str

    @property
    def passage_id(self) -> str:
        key = "\x1f".join((self.source_id, self.revision, *self.heading_path, str(self.ordinal)))
        return sha256(key.encode("utf-8")).hexdigest()

p = Passage("docs/widget", "sha256:abc", ("Limits", "Retries"), 0, "Retry at most twice.")
print(p.passage_id, p.heading_path)

Exercise

Design an index manifest

Turn a versioned documentation site with code, tables, and access groups into a searchable corpus.

  1. Define identities, parsing rules, unit boundaries, fields, and analyzer choices.
  2. Specify atomic publication, rollback, and deletion propagation.
  3. Create pre-publication data-quality checks and an incomplete-ingestion policy.

Success criteria

  • A result resolves to an immutable source revision.
  • Permission changes cannot wait for a full rebuild.
  • Parser and model upgrades never create an unlabelled mixed generation.

Reflect: Which source structure would be most damaged by naive fixed-token chunks?

References and further reading