Skip to content

Chapter 4.5 — Memory poisoning and vector poisoning

🎯 Objective

Treat memory poisoning and vector poisoning as operational risks with possible mitigation, but requiring discipline in ingestion, provenance and lifecycle.

🧠 Concept

  • Memory poisoning. Incorrect, malicious or ambiguous data is persisted in the agent's memory (preference, fact, profile) and reused in future interactions. It can come from user input, from the model's own output (auto-poisoning), from tool output or from a retrieved document.
  • Vector poisoning. A malicious or noisy document enters the RAG index. In future retrievals, the contaminated passage becomes context and influences answers — including triggering indirect prompt injection.
  • Training data poisoning. A more well-known variant in traditional ML: the training dataset is contaminated to induce a specific behavior in the model. Outside the direct scope of this chapter (see Ch. 4.9 — supply chain).

🛡️ Mitigations

Principles common to both vectors:

  • An ingestion pipeline with validation. Schema, content, sensitivity classification.
  • Sanitization of known payloads. Removal of hostile markup, control characters, explicit imperative instructions.
  • Provenance. Each document/memory carries origin, author, timestamp, hash. Memory without provenance is not trustworthy.
  • Approval for new sources. Adding a new ingestion source requires review, not just configuration.
  • Per-tenant isolation. Isolated memory and indexes; cross-tenant is an entire class of risk eliminated by design.
  • Adversarial evals. A set of "hostile documents" in the retrieval and agent eval set.
  • TTL and deletion. Memory is not eternal; preferences expire.
  • Right to be forgotten implemented. Not just conceptually.

Specific to memory:

  • Classify the type: confirmed fact, inference, model guess. Persist only the first.
  • Require confirmation for persistent preferences (explicit UX).
  • Allow editing and deletion by the user.

Specific to the vector store:

  • Fast reindexing when a document is deemed contaminated.
  • Quarantine. Suspicious documents leave the production index before investigation.
  • EX-SEC-04 — a vector poisoning simulation with sanitization + reindex.

📚 References