Operate RAG and prove mastery

Production RAG is a continuously changing evidence service. Sources publish revisions, permissions change, index generations roll, models drift, queries shift, and incidents expose unknown failure modes. Operations must preserve a chain from authoritative source to returned claim while meeting latency, cost, availability, privacy, and security objectives. A fluent answer is only one artifact of that chain.

Partition the architecture into owned services. Source connectors and parsers build evidence units. Index builders publish lexical and vector generations. A query service resolves policy and retrieves candidates. A context service constructs packets. A generation service invokes models without ambient authority. A verifier assesses claims and citations. An evidence store retains governed run artifacts. Each boundary has authenticated identities, schemas, deadlines, and owners. Smaller systems may combine deployments, but should preserve logical contracts.

Version the complete release tuple: source watermarks, parser, chunking, analyzers, embedding model and input, vector index, lexical index, query planner, fusion, reranker, context policy, generator, prompt policy, output schema, verifier, and evaluator. A response records the tuple or a release ID that resolves to it. This is the minimum information needed to compare behavior and investigate regressions.

Rollouts begin offline on fixed corpora and cases, then shadow on protected production-like traffic, canary by low-risk slice, and expand through gates. Compare baseline and candidate on critical invariants, retrieval and claim metrics, abstention, security tests, p95 and p99 latency, cost, and error reasons. Model upgrades can change citation behavior even when retrieval is identical. Index and model rollouts should be independently reversible unless their versions are coupled.

Freshness and deletion need source-to-answer SLOs. Track source event time, ingestion, parsing, embedding, index promotion, replica watermark, cache invalidation, and query visibility. Urgent permission revocation takes a fast masking path and later physical cleanup. Test that a deleted unit cannot appear in candidates, packets, responses, caches, or new evaluation artifacts. Historical evidence retention follows legal and incident policy rather than ordinary search availability.

Observe stages with traces, metrics, and protected logs. A run trace records request and release identity, policy decision reference, plan branches, adapter statuses, candidate counts and IDs under controlled access, context selection, model call metadata, claim validation, terminal reason, cost, and latency. OpenTelemetry semantic conventions provide a consistent base, but avoid unbounded metric labels and raw sensitive content. Sample full traces by risk, anomaly, and consent policy.

Define SLOs beyond uptime: authorized-answer rate, verified-answer rate, correct-abstention rate, corpus freshness, urgent-revocation latency, index coverage, retrieval candidate recall on probes, claim-support rate on sampled cases, unknown-citation count, degraded-route rate, cost per verified answer, and cleanup completeness. Some quality measures arrive slowly. Use canary probes and leading indicators without claiming they replace judgments.

Capacity budgets include embedding throughput, index build headroom, vector memory, search fan-out, reranker batches, context tokens, generator throughput, verifier calls, evidence storage, and evaluation capacity. Admission control should reject or queue work before deadlines become impossible. Cache query embeddings and safe retrieval stages with complete scope keys. Do not cache final answers across changing evidence or policy without generation-aware validation.

Prepare runbooks for source compromise, poisoned generation, stale permissions, index corruption, embedding regression, generator outage, verifier outage, provider data-policy change, runaway adaptive loops, and unsupported-claim reports. Kill switches should disable an external corpus, model, query route, or generation. Recovery includes fixed probes, policy tests, citation resolution, and trace inspection. Run game days and convert discoveries into tests.

The mastery capstone is an ontology-aware technical-support agent over a versioned corpus. Build deterministic ingestion with lineage and access tags; lexical and dense retrieval with fusion; context coverage and token budgets; structured claims and citations; bounded corrective retrieval; independent verification; and a harness that records evidence, budgets, and faults. The ontology should define product, version, component, symptom, cause, and remediation relations and be used for scope or facet coverage without replacing textual evidence.

Evaluate the capstone on direct facts, paraphrases, identifiers, multi-hop dependencies, version conflicts, missing evidence, restricted evidence, stale content, injected instructions, poisoned passages, and service faults. Report corpus, candidate, context, claim, abstention, security, latency, and cost metrics. Include exact-versus-ANN tests where applicable and multiple generation trials. Demonstrate a deletion, a model rollback, a retrieval timeout fallback, and a failed citation.

Acceptance evidence should let an independent reviewer replay the deterministic stages from an immutable fixture, inspect every returned claim’s sources, verify that budgets and policies held, and reproduce evaluation. Document remaining limitations: unsupported languages, source gaps, judge uncertainty, model-provider dependencies, and residual injection risk. Do not claim production readiness from a polished demonstration.

Operational maturity means the system can say what it knew, why it retrieved it, which evidence reached the model, which claims were supported, what it declined, and how the release behaved under failure. That transparency enables improvement and accountable use.