Skip to content

Chapter 4.4 — Tool misuse and excessive agency

🎯 Objective

Treat tool misuse and excessive agency as architectural risks, not as prompt bugs. Most serious incidents with agents in production start here.

🧠 Concept

  • Tool misuse. The agent calls a tool in a way that is legitimate from the schema's point of view, but wrong from the business's point of view: invalid arguments, wrong resource, wrong moment, wrong tenant.
  • Excessive agency. The agent can do more things than it needs for the current task. The larger the surface, the higher the risk of tool misuse.
  • Confused deputy. The agent, acting with its own privileges, executes an action the user did not have permission to request.

🛡️ Mitigations

In layers, from the outside in:

  • Least privilege in the agent's identity (Ch. 4.8). Short tokens, minimal scopes, no reused human credentials.
  • An explicit per-role allowlist of tools in the harness.
  • Schema validation of the arguments before any action.
  • Policy-as-code (Ch. 4.7) deciding, at runtime, whether that tool call is allowed, given role, tenant, resource, risk and context.
  • Idempotency of state-modifying tools: calling twice with the same idempotency_key does not duplicate the effect.
  • Rate limits per tool, per agent, per tenant.
  • Budgets per tenant and per agent: cost and number of calls.
  • Sandbox for executing generated code.
  • HITL for high-risk actions (define objective criteria: financial value, change to a critical resource, external contact, etc.).
  • Audit trail linking action -> token -> policy decision -> result.

🚨 Operational signals to monitor

  • The rate of tool calls denied by policy (it should be >0; if it is zero, the policy is not being tested).
  • Attempts to use a tool outside the allowlist (a symptom of regression or attack).
  • A spike in calls to sensitive tools in short windows.
  • A sudden increase in cost per agent or per tenant.
  • EX-SEC-03 — a tool misuse simulation with a policy blocking the action.
  • EX-AGT-06 — an agent with a Rego policy sidecar blocking tool calls.