Implement a small RAG system from first principles

Building a small system without a framework reveals the contracts that libraries often hide. The goal is not to reproduce a production embedding model in a few lines. It is to implement the control plane around retrieval: immutable evidence units, query analysis, scoring, context budgets, citations, abstention, validation, and trace evidence. Once these boundaries are clear, production adapters can replace individual algorithms.

Define immutable inputs. An evidence unit has ID, source revision, text, authority, validity interval, and access tags. A request has actor scope, question, time, and answer policy. The pipeline never mutates the corpus during a query. It produces a run record containing versioned configuration and stage outputs. This makes test fixtures cheap and replay deterministic.

The runnable example below uses term overlap rather than a neural retriever. That limitation is intentional: relevance scoring is visible, and the surrounding behavior is the same for BM25 or vector adapters. The retriever filters authorization and validity before scoring. It returns identities and scores rather than anonymous strings. The context builder caps both item and word count and assigns handles.

Generation is represented by a deterministic synthesis function that copies a selected sentence. A real language-model adapter would receive the question, evidence packet, structured output schema, and deadline. Its output must still pass the same validator. Separating the adapter keeps tests independent of network access and allows recorded model fixtures.

The pipeline abstains if no positive-scoring evidence survives. This avoids the “nearest neighbor must be relevant” trap. A real threshold should be calibrated on held-out data and may incorporate reranker or support grades. The answer includes a claim and handle, not a fabricated source URL. Resolution to source metadata happens after validation.

Trace evidence should include request ID, corpus generation, policy version, retriever version, candidates before and after filters, scores, context omissions, model invocation identity, output, validation results, and terminal reason. Hash or protect sensitive values according to policy. A trace is not a chain-of-thought; it is an externally inspectable record of system transitions and artifacts.

Test invariants first. A denied unit never appears in candidates, context, output, or logs. An expired unit cannot support a current answer. Unknown citations fail. Empty evidence abstains. Context budgets are never exceeded. Tie ordering is deterministic. A corpus generation mismatch stops rather than mixing evidence. These properties matter more than whether a demo answer sounds polished.

Then test relevance. Construct queries for exact terms, paraphrases, ambiguous entities, negation, and unsupported facts. The toy overlap scorer will fail paraphrases, producing a clear reason to add dense retrieval. Add a second adapter and RRF without changing the evidence contract. Add a reranker without changing policy. This incremental path turns architecture choices into measured interventions.

Concurrency adds deadlines and cancellation. Run independent retrievers in parallel with bounded output. Reserve budget for context, generation, and validation. If one adapter fails, return a declared degraded result only where policy allows. Idempotent request identities can prevent client retries from producing duplicate billable generations or conflicting artifacts.

Persist only what the product needs. A stateless query service can return the complete run artifact to an evidence store. Conversations require user-visible state and explicit provenance for prior claims. Caches key on corpus generation, policy scope, retrieval configuration, model version, and answer schema. Omitting any of those can yield stale or cross-scope answers.

Frameworks can accelerate connectors, model calls, and orchestration, but they do not remove these responsibilities. Inspect whether a framework preserves stable IDs, exposes raw retrieval evidence, enforces policy before model context, supports typed terminal states, and allows stage replay. Wrap provider objects at your boundary so upgrades do not rewrite the entire system.

The mastery outcome is a system whose retrieval algorithm can be swapped while tests for authority, provenance, citations, abstention, and budgets remain unchanged. That is the difference between a RAG demo and an engineered evidence application.