Skip to content

Chapter 4.2 — Threat modeling for ML, LLMs and agents

🎯 Objective

Adapt STRIDE / PASTA for AI systems, integrating the vocabulary of OWASP LLM Top 10, NIST AI RMF, the NIST GenAI Profile and MITRE ATLAS. The goal is not to cover all the theory; it is to give the architect a set of questions that cannot be forgotten in a design review.

🧠 STRIDE adapted to AI systems

Category Typical AI attack Where to mitigate
Spoofing An agent impersonates a user; a user impersonates an agent; another tenant's token Strong identity (Ch. 4.8), audit trail, on-behalf-of
Tampering Manipulation of a dataset, RAG index, memory, model artifact, policy bundle Data contracts, hashes, signing, supply chain (Ch. 4.9)
Repudiation An action executed without a trail; user/agent denies it Immutable audit log, attribution via token + policy decision
Information disclosure PII leak via prompt, log, response; cross-tenant via RAG DLP, redaction, tenant filter in storage, limited retention
Denial of service Agent loop; token explosion; retrieval abuse Budgets, circuit breakers, max steps, per-tenant rate limit
Elevation of privilege Agent runs a tool outside scope; tool description deceives the model Per-role allowlist, policy, HITL, minimal scopes

🧠 OWASP LLM Top 10 — quick mapping

Most items of the OWASP LLM Top 10 correspond to specific chapters of this book:

OWASP LLM Chapter of this book
LLM01 — Prompt Injection (direct and indirect) Ch. 4.3
LLM02 — Insecure Output Handling Ch. 2.4, Ch. 4.7
LLM03 — Training Data Poisoning Ch. 4.5 (vector poisoning), Ch. 4.9 (supply chain)
LLM04 — Model Denial of Service Ch. 6.2, Ch. 6.5
LLM05 — Supply Chain Vulnerabilities Ch. 4.9
LLM06 — Sensitive Information Disclosure Ch. 4.6
LLM07 — Insecure Plugin / Tool Design Ch. 3.5, Ch. 3.7, Ch. 4.4
LLM08 — Excessive Agency Ch. 3.2, Ch. 4.4
LLM09 — Overreliance Ch. 5.6, and the book's central thesis
LLM10 — Model Theft Ch. 4.9, Ch. 4.11

🧠 Reference frameworks

  • OWASP Top 10 for LLM Applications. A practical, developer-oriented catalog.
  • NIST AI Risk Management Framework 1.0. An organizational structure for risk governance.
  • NIST Generative AI Profile. A cut of the RMF for generative AI.
  • MITRE ATLAS. A taxonomy of tactics and techniques for attacks on ML.

🧪 How to apply it in an architecture review

In a design review, go through the six verbs of adapted STRIDE and, for each system component, answer:

  1. Who talks to whom? A diagram of trust boundaries.
  2. What data crosses which boundary? Includes PII, secrets, tenant content.
  3. What is the control, what is the mitigation? Mark explicitly.
  4. What is the residual risk? If unaccepted risk remains, stop and re-architect.

🏢 Hermes Logística — wave 4

The first serious security review at Hermes began when the team tried to map "what happens if the FAQ document contains IGNORE INSTRUCTIONS ABOVE". The discussion produced three things: a threat model in adapted STRIDE, a list of five deterministic controls (tenant filter in storage, schema validation, Rego policy, egress allowlist, HITL for high risk) and an accepted-residual-risk sheet. It was the first time security stopped being "let's add a guardrail" and became part of the architecture.

📚 References