Knowledge-agent threat model and fail-closed design

Knowledge agents combine untrusted content, powerful interpretation, credentials, memory, and side effects. Their threat surface spans data ingestion, retrieval, prompts, model proposals, tool adapters, external systems, event logs, and long-term memory. Security begins with assets and trust boundaries: user data, source evidence, credentials, graph structure, actions, approvals, audit records, and availability.

Indirect prompt injection places malicious instructions in documents, web pages, tool output, graph labels, tickets, or memories that the agent later retrieves. The defense is architectural. Mark external content as data, keep system policy outside retrieved context, expose least-privilege typed tools, validate actions independently, and require approvals for effects. Asking the model to “ignore malicious instructions” is useful context but not an enforcement boundary.

Data poisoning seeks durable influence rather than one prompt. An attacker can create false claims, repeated narratives, deceptive aliases, high-centrality edges, or memories that bias future plans. Preserve provenance, separate generated from authoritative claims, cap influence by source, validate identity merges, monitor graph drift, and require elevated review for claims with large downstream blast radius.

Authorization failures include confused deputy behavior: the model uses a service credential to perform an operation the user could not. Carry the initiating identity and delegated scope through every retrieval and action. Filter before ranking and summarization so hidden content cannot influence visible results. Test graph inference and aggregates for indirect disclosure. Relationship-based policy must be evaluated at the resource version used for the action.

Tool abuse includes argument smuggling, path traversal, command injection, server-side request forgery, excessive queries, and exfiltration through output channels. Narrow schemas, allow lists, parameterized operations, sandboxes, network egress controls, output limits, and recipient policies address these at the adapter layer. Never expose a general shell or unrestricted browser when a bounded domain operation suffices.

Memory creates cross-task persistence. A poisoned task can attempt to store instructions, secrets, or false facts for another user. Memory publication requires tenant and subject binding, evidence, type, expiry, sensitivity classification, and policy. Retrieval occurs inside the current authorization scope. Users need mechanisms to inspect, correct, and delete applicable memory.

Denial of wallet and service can exploit loops, expensive global retrieval, recursive graph paths, huge tool results, or forced retries. Enforce budgets and complexity limits before execution, rate-limit by actor and tenant, cap traversal and context, and detect no-progress states. A graceful terminal response is safer than exhausting infrastructure while trying to be helpful.

Audit and observability can leak the very content they protect. Log stable IDs, decisions, hashes, sizes, timings, and redacted summaries by default. Restrict raw prompt and source capture. Sign or protect high-value audit records and separate operators who can act from those who can alter logs. Detect unusual approval reuse, recipient expansion, policy denial spikes, and graph influence changes.

Fail closed when identity, authorization, schema, evidence, target version, or effect outcome is ambiguous. Fail usable by returning a typed reason, preserving completed safe work, and offering an approved recovery path. A security denial should not be translated into a fabricated no-data answer; distinguish “no authorized evidence” from “no evidence exists.”

Red-team the system as trajectories, not isolated prompts. Attempt retrieval injection followed by tool use, cross-tenant entity resolution, poisoned memory, stale approval replay, ambiguous write retry, graph path exfiltration, and budget exhaustion. Record the first control that should stop each attack and add regression tests. Use the NIST Generative AI Profile to organize risks and governance, while mapping controls to the concrete architecture and consequences of this application.