Skip to content

Chapter 3.2 — Agency, limited autonomy and blast radius

🎯 Objective

Establish that autonomy is not an emergent property: it is a function of the action space, the available tools, the granted permissions and the applied policies. Whoever designs the agent designs the ceiling of autonomy, and therefore the ceiling of risk.

🧠 Concepts

  • Agency. The operational capacity to pursue a goal via intermediate decisions and permitted actions.
  • Limited autonomy. Agency constrained by scope, available tools, IAM permissions, cost budgets, a maximum number of steps and human approvals at critical points.
  • Blast radius. How much a wrong action by the agent can damage the system/company. Includes reversible effect, permanent effect, number of tenants affected, financial value exposed.

🛡️ Principles

  • Fewer permissions > more permissions. Increasing privilege is a conscious decision, with review; reducing is the default.
  • Human approval for irreversible actions. Objective criteria (value, action class, affected resource) declared before go-live.
  • Budget + circuit breaker to limit cost and number of steps.
  • Observability requires tracing of each decision, not just the final result.
  • Defense in depth across identity, policy, schema and tool registry — no isolated layer.

🚨 Signs of excessive agency

  • An agent with access to tools it "does not need yet", but "might come to need".
  • An agent sharing a service credential with a human or with another agent.
  • An agent without a clear max_steps.
  • An agent without a per-tenant budget.
  • An agent whose only defense against misuse is the prompt.

📚 References