Skip to content

Chapter 4.1 — Security principles in AI

🎯 Objective

Establish the operational premises and principles that structure the rest of Part 4. Without this foundation, it is easy to confuse mitigation with control, and prompt with policy — the two most expensive mistakes in AI security.

🧠 Operational premises

Treat as true until proven otherwise:

  • Inputs are untrusted. Any text coming from the user, the document, the tool or the neighboring agent may be trying to influence the system.
  • Retrieved documents may contain malicious instructions. Indirect prompt injection is a real vector (Ch. 4.3).
  • Tool outputs may contain hostile payloads. A tool's result may include markdown, HTML, JSON or text that tries to manipulate the model in the next iteration.
  • Memory can be contaminated. Persisted preferences derive from inputs — see Ch. 4.5.
  • Tools can cause real, irreversible effects. Actions in the world (email, payment, CRM change) are one-way.
  • Agents can amplify privileges without realizing it. The classic confused deputy, with more agency than it should have.
  • Logs can leak sensitive data if not redacted.

🧠 Principles

  • Defense in depth. Redundant layers; no single line.
  • Least privilege. The minimum permission that still works.
  • Zero trust between components. Includes agents, tools, MCP servers, A2A peers.
  • Tenant isolation. Real, not nominal.
  • Auditability. Every executed action needs a trail.
  • Reversibility whenever possible; when not, require human-in-the-loop.
  • Privacy by design. Minimization from collection onward, not just in the final log.
  • Deterministic control precedes probabilistic mitigation (Ch. 0.6). Critical security goes in code.

🧠 The difference between control, mitigation and neither

Recapping what is in Ch. 0.6, now applied to security:

Type Example What it guarantees
Deterministic control RBAC, schema validation, policy-as-code, allowlist, egress block Predictable behavior, if well implemented
Probabilistic mitigation Input sanitization, heuristic injection detector, risk classifier Reduces risk, does not eliminate it
Neither one nor the other A prompt saying "do not do X", a tool description "use with care" Nothing

Every risk analysis in AI must name, for each control, which of the three categories it is. Treating a mitigation as a control is the classic path to an incident.