Threat-model retrieved content and generated action

RAG joins untrusted or differently trusted content with a language model, making the retrieval boundary a security boundary. Retrieved text can influence generated output even when it is presented as reference material. Indirect prompt injection exploits this property: an attacker places instructions in data the model later consumes. PoisonedRAG studies knowledge-corruption attacks that add malicious texts to a retrieval database so target questions produce attacker-chosen answers. Security must therefore cover ingestion, retrieval, model context, tools, output, and evidence stores.

Start with assets and trust zones. Assets include private corpus text, query content, embeddings, access metadata, credentials, model context, generated answers, citations, tools, logs, and evaluation data. Actors include source owners, users, tenants, administrators, model and embedding providers, and attackers who can influence public or internal documents. Trust is purpose-specific: a document may be readable but not authoritative, and authoritative prose can still contain strings that resemble instructions.

Indirect injection cannot be solved by telling the model to ignore instructions in documents. Delimit and label evidence, minimize included text, remove active markup where appropriate, and use models without unnecessary tools. Most importantly, enforce capability and output policy outside the model. Retrieved content cannot grant a tool, change tenant, approve a transaction, or override a system policy. High-consequence actions require typed proposals, authorization, validation, and often human confirmation.

Corpus poisoning defenses begin with provenance. Authenticate ingestion sources and publishers, separate trust classes, restrict who can set authority metadata, scan unexpected changes, detect duplicate or keyword-stuffed content, and canary index generations. Stable lineage lets responders find every derived unit from a compromised source. Similarity and popularity do not establish authority. When external search is allowed, label it distinctly and avoid using it for decisions requiring governed sources.

Enforce authorization before model context. Tenant, classification, purpose, residency, and time filters apply during candidate generation where possible and again before context construction. The generator provider must be authorized to process the selected data. Do not log denied candidate text or expose hidden result counts. Cache keys include policy scope and corpus generation. Treat embeddings as sensitive derived data, not anonymized content.

Query manipulation includes prompt injection by the user, filter injection, resource-exhaustion requests, and attempts to retrieve secrets. Parse filters through a schema and server-side allow-list. Cap query length, branches, candidates, context, model tokens, calls, and time. Detect repeated or adversarial decompositions. Rate-limit by authenticated principal and consequence, not only IP.

Citations create another attack surface. A generator can fabricate a trusted-looking URL, cite an evidence handle that does not support the claim, or choose a low-authority source. Allow only server-generated handle resolution. Validate locator domains and avoid rendering active HTML from titles or snippets. Display revision and authority so the user can distinguish sources. Link handling should not leak private query or tenant tokens.

Output can contain secrets copied from authorized context but inappropriate for the response audience, malicious code, or unsafe procedures. Apply purpose-aware data-loss prevention, content policy, and structured validators. Authorization to read a source does not automatically authorize broad redistribution. For code or commands, present them inertly unless a separate execution harness validates and authorizes effects.

Evaluation and red teaming must cover targeted attacks. Seed documents with instruction-like text, counterfeit authority claims, target query phrases, hidden Unicode, malicious links, and conflicting revisions. Test cross-tenant caches, stale permissions, deleted documents, external-source broadening, citation spoofing, and judge manipulation. Measure attack success, unauthorized disclosure, harmful action proposal, and detection—not merely answer quality on benign queries.

Incident response requires fast containment by source, revision, corpus generation, tenant, model, prompt policy, and external connector. Quarantine compromised units, invalidate caches, switch aliases, and identify affected runs through evidence IDs. Preserve enough protected evidence for investigation. Notify source owners and users according to policy. Add the attack to a regression corpus without retaining prohibited sensitive content.

NIST’s Generative AI Profile provides a risk-management frame across governance, content provenance, testing, incident disclosure, and security. Apply it to the complete application rather than treating the language model as the only component. The core security principle is non-interference: untrusted retrieved text may inform a claim, but it cannot silently change authority, policy, capabilities, or the meaning of success.