Embeddings & Hybrid Search

Retrieval is the controlled reduction of a large collection to a small set worth inspecting. Embeddings, inverted indexes, and vector databases are mechanisms; the product requirement is a relevance contract. That contract states which information need the system serves, what counts as useful evidence, how quickly results must arrive, which documents the caller may see, and how mistakes will be measured. Without it, teams optimize an attractive similarity score while users experience missed facts, stale policies, or unauthorized results.

Begin with the distinction between a query and an information need. A query is the observable string or object submitted by a caller. The information need is the latent goal: find a known document, compare alternatives, answer a fact, investigate an incident, discover adjacent concepts, or gather all records that satisfy a condition. The same text can express different needs. “Mercury limits” could request an environmental regulation, a database product setting, or facts about a planet. Relevance therefore depends on user, task, time, jurisdiction, and corpus—not only semantic resemblance.

A retrieval contract should name the unit of judgment. Binary relevance asks whether a document is useful at all. Graded relevance distinguishes decisive evidence, supporting evidence, background, and distraction. Passage-level labels may reward a precise chunk even when its parent document is only loosely related. Set-level judgment asks whether the returned group covers distinct facets without repetition. Temporal judgment asks whether the result was valid at the query time. Security judgment is non-negotiable: a highly relevant result that violates access policy is a system failure, not a quality trade-off.

Separate candidate generation from ranking. Candidate generation searches cheaply and broadly enough to preserve likely evidence. Ranking spends more computation to order a much smaller set. A hybrid system may generate candidates from a lexical inverted index and a dense vector index, union their identities, fuse or normalize scores, apply structured filters, rerank with a cross-encoder, diversify the set, and return evidence with provenance. Each stage has its own recall, latency, and failure modes. Evaluating only the final list hides whether a miss originated in ingestion, candidate generation, filtering, fusion, or reranking.

Lexical and semantic retrieval provide complementary signals. Lexical matching is excellent for exact identifiers, rare names, quoted phrases, and domain terminology. Dense embeddings can connect paraphrases and concepts that share few surface words. Learned sparse models retain interpretable token dimensions while expanding terms. Late-interaction models preserve token-level structure. None dominates every corpus. The right architecture follows error analysis: measure which query slices each method rescues, not which technique is fashionable.

Formalize a result as an evidence record rather than a text string. It should carry stable document and passage identifiers, source version, fields used for scoring, raw component scores, rank, access decision, timestamps, and a locator back to the canonical source. Preserve raw scores even when user-facing scores are normalized. This record makes debugging possible and allows downstream systems to cite, deduplicate, filter, and audit results.

Three scales clarify the contract. In a toy recipe collection, “no-bake lemon dessert” needs semantic matching, but an allergy filter is a hard constraint. In a developer portal, exact error codes should trigger lexical matches while descriptive symptoms benefit from embeddings. In an enterprise discovery service, many tenants, languages, document versions, and legal holds require authorization before scoring, freshness guarantees, and per-domain evaluation. The algorithm may be similar, but operational consequence changes the design.

Do not define success as “the expected answer appears somewhere.” Choose metrics tied to the experience. Recall at k measures whether relevant evidence survived candidate generation. Mean reciprocal rank rewards placing the first relevant item early. Normalized discounted cumulative gain handles graded relevance and position. Precision measures distraction. Coverage measures whether required facets appear. Latency, freshness, index completeness, denial correctness, and cost are parallel dimensions. A release gate can require minimum recall on critical slices before optimizing average rank.

The remainder of this book develops the contract into a system: corpus modeling, lexical retrieval, embedding geometry, approximate nearest-neighbor indexes, hybrid fusion, reranking, an end-to-end implementation, evaluation, and production operations. At every layer, retain stable identities and stage-level evidence. Retrieval quality is not a mysterious property of a model. It is an observable consequence of data, representations, algorithms, policies, and operating conditions.