AI Agents Need Budgets, Not Just Tokens

Giving AI agents access to production systems is becoming increasingly common, but simply handing over a "token" with a list of tools can lead to unintended consequences, as demonstrated by an incident where an AI agent accidentally deleted hundreds of workloads. Sachin Malhotra, an engineer on the CI team at Anthropic, shared his insights at AI Engineer World's Fair on how to manage AI agents more safely by giving them "budgets" rather than unlimited "tokens."

The Perils of Unbounded Access

Malhotra recounted an incident where an AI agent, tasked with cleaning up unused workloads, misidentified its targets due to a faulty filter. This led to the deletion of approximately 200 workloads, impacting 20 engineers and potentially erasing hours of work. The agent, acting on its programming, genuinely believed it was performing a necessary cleanup. This scenario highlights a critical gap: the common practice of granting agents broad access via tokens and tool lists is insufficient when these agents begin performing real-world operations in production.

The problem, Malhotra explained, is that tokens are often treated as a boolean yes/no permission. If the scope is too narrow, the agent becomes useless; if it's too wide, the risk of catastrophic errors increases. This is akin to onboarding a junior engineer without proper guidance or guardrails. Unlike humans, AI agents don't tire, but they can be "confidently wrong."

Introducing the Budget: A New Framework for AI Control

Malhotra proposed a shift from "tokens" to "budgets" for managing AI agent capabilities. A budget, he elaborated, is not a single number but a multi-dimensional concept encompassing:

• How much an agent can do.
• How fast it can do it.
• What it can undo on its own.
• Who is observing its actions.

He then outlined three key primitives and one lens to help implement this budget-based approach:

Primitive 1: Asymmetric Verbs

Verbs, or actions an agent can take, can have vastly different blast radii depending on whether they succeed or fail. Malhotra categorized these as:

• Fails Loud: Actions like unskipping a test or paging a human. If they fail, the error is immediately apparent (e.g., CI builds turn red, or an on-call engineer is unnecessarily disturbed). These failures are easily corrected.
• Fails Silent: Actions like skipping a test incorrectly. The failure might go unnoticed until much later, potentially allowing real bugs into production without immediate detection.

The recommendation is to grant agents access to verbs that "fail out loud" on a dashboard, while requiring human intervention for verbs that can fail silently.

Primitive 2: Rate Limits

Rate limits provide a concrete form of budget, acting as a ceiling on disruptive actions within a given time window. Each caller receives a limited number of actions, with autonomy within that limit. If the limit is exceeded, the request is denied, and the agent must wait for the limit to refill. Crucially, rate limits can be scaled based on context, such as the namespace of the resources being accessed. Malhotra highlighted an example where a rate limit on deletions was implemented via a webhook, preventing the previous incident. The bypass flag in such systems should ideally prompt the agent to ask a human to perform the action directly.

Primitive 3: Tripwires Over Allow Lists

While allow lists are static guesses about an agent's needs, tripwires provide a mechanism to gather data on agent behavior after the fact. For cheaper actions, agents can operate freely, with their actions logged and monitored. Tripwires act as "smoke detectors," alerting humans when aggregate behavior deviates from the norm (e.g., an unusually high number of investigation threads launched by an agent).

Malhotra shared an example where a tripwire detected an excessive number of investigation threads, revealing an underlying infrastructure issue. The fix involved a simple prompt adjustment, guiding the agent to correlate failures before launching individual investigations. This iterative improvement is a key advantage of tripwires over static allow lists.

The Lens: The Undo Test

The "undo test" serves as a critical lens for evaluating the safety of agent actions. It poses two questions: Can the agent undo its own action, and if it gets it wrong, how bad is the impact? If an agent can self-correct and the potential impact is acceptable, the action can be logged and allowed. If either answer is no, a "second key" or human oversight is required.

Policy as Text and Infrastructure

Malhotra emphasized that policy for AI agents needs to exist in two layers: text and infrastructure.

• Text: Prompts and context files shape the agent's intent and provide reasoning, but they are advisory and lack enforcement.
• Infrastructure: This includes proxy layers and admission webhooks that enforce limits and budgets deterministically, bounding how wrong an agent can go.

He illustrated this with a diagram showing how each agent session has a proxy that stamps its identity onto every outbound call. This identity is then used by various safeguards throughout the system, ensuring accountability and control without the agent itself needing to manage its provenance.

Key Takeaways

Malhotra summarized his core message with four key points:

• Asymmetric Verbs: Grant agents access to actions that "fail out loud" and keep humans involved for actions that "fail silently."
• Rate Limits: Implement ceilings on every write operation that can refill autonomously, eliminating the need for ticket filings.
• Tripwires: Monitor aggregate behavior rather than individual calls and use simple text-based fixes to guide agent actions.
• Undo Test: Use this as a sizing tool for all primitives, asking if the agent can recover from its actions and what the impact of failure would be.

Underpinning all these principles is the idea that identity must come from the infrastructure, not the request itself. Getting this foundational rule right, Malhotra concluded, simplifies the process of safely integrating AI agents into production workflows.