Reading tools and contents
4 min
Retrieval Engineering

Chapter 8 of 10

Implement a retrieval pipeline with explicit stages

From first principles

About 4 minutes · includes examples, an exercise, and references

Chapter at a glance

  • •Typed stages isolate policies, algorithms, failures, and upgrades.
  • •Authorization predicates are resolved server-side and verified again before text return.
  • •Deadlines, generations, and degraded coverage are part of the response contract.

A maintainable retrieval service is a sequence of typed stages rather than one opaque search call. The stages are query validation, policy context resolution, query analysis, route selection, lexical and semantic candidate generation, identity union, fusion, reranking, diversification, presentation, and evidence recording. Not every query uses every stage, but every route should produce the same result contract and terminal status.

Define input before implementation. A search request can include query text, tenant, actor, purpose, language hint, filters, requested k, deadline, and trace context. Validate size and allowed filter fields. Resolve actor permissions into server-side predicates; do not trust a tenant or access group merely because the client supplied it. Normalize request time once so temporal filters and logs agree.

Query analysis should preserve the original string and produce an immutable analysis object. It may identify quoted phrases, identifiers, language, spell-correction candidates, and intent features. Corrections are hypotheses, not replacements; retain the original route and consider searching both when the error cost warrants it. Version analyzers and synonym sets because they affect reproducibility.

Adapters isolate engine-specific behavior. A lexical adapter accepts a structured query and returns candidate records. A vector adapter accepts a vector plus compatible filter and generation. Both enforce deadlines and output caps. They return typed errors for timeout, unavailable shard, invalid filter, incompatible generation, or partial coverage. The coordinator decides fallback; adapters should not silently broaden policy or switch data generations.

Stable identities make merging deterministic. Use a map keyed by passage ID and append component evidence. If two candidates claim the same ID but different revision or tenant, treat it as an integrity error. Fusion produces a new score without overwriting raw evidence. Reranking consumes only the capped union. Presentation joins authoritative metadata by identity and verifies policy again before returning text; defense in depth protects against stale caches and adapter mistakes.

The compact runnable example uses supplied lexical and dense rankings so orchestration is visible without third-party packages. It validates limits, fuses with RRF, applies an allow-list before and after ranking, diversifies by parent, and returns an evidence structure. Production adapters replace the lists, but the contracts remain testable in memory.

Deadlines should be propagated as remaining time. Parallel candidate calls receive a share of the request budget. Reserve time for merge, rerank, and serialization. On one retriever timeout, either return an explicitly degraded result if policy permits or return an error. Include completeness and fallback reason in the response. Do not let a slow optional reranker cause the client to retry the entire request and multiply load.

Caching is stage-specific. Query embedding caches key on normalized encoding input and model version. Retrieval caches key on query, filter, authorization scope, index generation, and component parameters. Final response caches additionally include presentation and policy versions. A cache that omits tenant or generation can leak data or preserve deleted content. Use short lifetimes for mutable corpora and active invalidation for revocation.

Observability should answer how each item reached the list. Record stage timings, adapter status, generations, candidate counts, filter counts, component ranks, fusion contributions, rerank deltas, truncation, diversity removals, and returned identities. Avoid logging raw confidential queries by default; store protected content only under an explicit evidence policy. Metrics use bounded labels—model version and status, not passage ID or raw query.

Testing follows boundaries. Unit-test analyzer rules, score calculations, tie-breaking, filter intersection, and response serialization. Contract-test adapters against recorded fixtures. Property-test invariants such as “a denied passage is never returned” and “adding an unrelated unauthorized document cannot change authorized results.” Integration tests use immutable index snapshots. Load tests include partial failures and selective filters. End-to-end tests join returned IDs back to source revisions.

The architectural payoff is replaceability. Teams can upgrade a vector model, ANN index, fusion rule, or reranker one stage at a time, replaying the same query set and examining stage deltas. A monolithic database call may be convenient initially, but an explicit evidence contract remains valuable even if one provider implements several stages.

Key points

  • Typed stages isolate policies, algorithms, failures, and upgrades.
  • Authorization predicates are resolved server-side and verified again before text return.
  • Deadlines, generations, and degraded coverage are part of the response contract.
  • Stage-specific evidence enables replay and component-level tests.

A deterministic hybrid coordinator

Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.

A deterministic hybrid coordinatorpython
from collections import defaultdict

def search(lexical, dense, allowed, parent, limit=3, rrf_k=60):
    if not 1 <= limit <= 20:
        raise ValueError("limit outside policy")
    scores, evidence = defaultdict(float), defaultdict(dict)
    for source, ranking in (("lexical", lexical), ("dense", dense)):
        unique_ids = [item for item in dict.fromkeys(ranking) if item in allowed]
        for rank, item in enumerate(unique_ids, 1):
            contribution = 1 / (rrf_k + rank)
            scores[item] += contribution
            evidence[item][source] = {"rank": rank, "contribution": contribution}
    ordered = sorted(scores, key=lambda item: (-scores[item], item))
    returned, seen_parents = [], set()
    for item in ordered:
        if item not in allowed:             # presentation-boundary recheck
            continue
        if parent[item] in seen_parents:
            continue
        returned.append({"id": item, "score": scores[item], "evidence": evidence[item]})
        seen_parents.add(parent[item])
        if len(returned) == limit:
            break
    return {"status": "complete", "results": returned}

print(search(["p1", "p2", "p3"], ["p3", "p4", "p1"],
             {"p1", "p3", "p4"}, {"p1":"d1", "p3":"d2", "p4":"d2"}))

Exercise

Build a contract-tested coordinator

Implement the pipeline around fake lexical, vector, and reranking adapters.

  1. Define request, candidate, component-evidence, and response types.
  2. Implement policy intersection, deadlines, fusion, fallback, and presentation recheck.
  3. Test denial, timeout, partial coverage, incompatible generations, ties, and deletion.

Success criteria

  • Unauthorized text cannot enter results, caches, or traces.
  • Every returned item has a reproducible stage ledger.
  • Adapter failure never causes an undeclared search broadening.

Reflect: Which hidden behavior in a current search SDK should become an explicit stage?

References and further reading