Artificial Intelligence

AI Agents Need Budgets, Not Just Tokens

Anthropic's Sachin Malhotra argues that AI agents in production need budgets, not just broad tokens, proposing primitives like asymmetric verbs, rate limits, and tripwires.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

5 min read

Sachin Malhotra presents on AI agent safety at AI Engineer World's Fair
Sachin Malhotra discusses secure AI agent deployment strategies.Credit: AI Engineer
Visual TL;DR
AI agents in productionDriver
increasingly common to give AI agents access to production systems
From the article 9+ mentionsThis scenario highlights a critical gap: the common practice of granting agents broad access via tokens and tool lists is insufficient when these agents begin performing real-world operations in production.
Broad token accessDriver
From the article 2 mentionsThis scenario highlights a critical gap: the common practice of granting agents broad access via tokens and tool lists is insufficient when these agents begin performing real-world operations in production.
Unintended deletionsOutcome
AI agent accidentally deleted hundreds of workloads due to faulty filter
From the article 3 mentionsThis led to the deletion of approximately 200 workloads, impacting 20 engineers and potentially erasing hours of work.
Need for budgetsCore
AI agents need budgets, not just broad tokens, for safer operation
From the article 8 mentionsWhile allow lists are static guesses about an agent's needs, tripwires provide a mechanism to gather data on agent behavior after the fact.
Asymmetric verbsContext
primitive for AI control, like 'propose delete' instead of 'delete'
From the article 3 mentionsAsymmetric Verbs: Grant agents access to actions that "fail out loud" and keep humans involved for actions that "fail silently."
Rate limitsContext
primitive to control agent actions, preventing rapid, widespread changes
From the article 7 mentionsRate limits provide a concrete form of budget, acting as a ceiling on disruptive actions within a given time window.
TripwiresContext
primitive for AI control, better than allow lists for dynamic environments
From the article 5 mentionsTripwires act as "smoke detectors," alerting humans when aggregate behavior deviates from the norm (e.g., an unusually high number of investigation threads launched by an agent).
Safer AI controlEffect
managing AI agents more safely by giving them budgets and specific primitives
From the articleThis identity is then used by various safeguards throughout the system, ensuring accountability and control without the agent itself needing to manage its provenance.
Contents(8)

Giving AI agents access to production systems is becoming increasingly common, but simply handing over a "token" with a list of tools can lead to unintended consequences, as demonstrated by an incident where an AI agent accidentally deleted hundreds of workloads. Sachin Malhotra, an engineer on the CI team at Anthropic, shared his insights at AI Engineer World's Fair on how to manage AI agents more safely by giving them "budgets" rather than unlimited "tokens."

AI Agents Need Budgets, Not Just Tokens - AI Engineer
AI Agents Need Budgets, Not Just Tokens, AI Engineer

The Perils of Unbounded Access

Malhotra recounted an incident where an AI agent, tasked with cleaning up unused workloads, misidentified its targets due to a faulty filter. This led to the deletion of approximately 200 workloads, impacting 20 engineers and potentially erasing hours of work. The agent, acting on its programming, genuinely believed it was performing a necessary cleanup. This scenario highlights a critical gap: the common practice of granting agents broad access via tokens and tool lists is insufficient when these agents begin performing real-world operations in production.

The problem, Malhotra explained, is that tokens are often treated as a boolean yes/no permission. If the scope is too narrow, the agent becomes useless; if it's too wide, the risk of catastrophic errors increases. This is akin to onboarding a junior engineer without proper guidance or guardrails. Unlike humans, AI agents don't tire, but they can be "confidently wrong."

Introducing the Budget: A New Framework for AI Control

Malhotra proposed a shift from "tokens" to "budgets" for managing AI agent capabilities. A budget, he elaborated, is not a single number but a multi-dimensional concept encompassing:

  • How much an agent can do.
  • How fast it can do it.
  • What it can undo on its own.
  • Who is observing its actions.

He then outlined three key primitives and one lens to help implement this budget-based approach:

Primitive 1: Asymmetric Verbs

Verbs, or actions an agent can take, can have vastly different blast radii depending on whether they succeed or fail. Malhotra categorized these as:

  • Fails Loud: Actions like unskipping a test or paging a human. If they fail, the error is immediately apparent (e.g., CI builds turn red, or an on-call engineer is unnecessarily disturbed). These failures are easily corrected.
  • Fails Silent: Actions like skipping a test incorrectly. The failure might go unnoticed until much later, potentially allowing real bugs into production without immediate detection.

The recommendation is to grant agents access to verbs that "fail out loud" on a dashboard, while requiring human intervention for verbs that can fail silently.

Primitive 2: Rate Limits

Rate limits provide a concrete form of budget, acting as a ceiling on disruptive actions within a given time window. Each caller receives a limited number of actions, with autonomy within that limit. If the limit is exceeded, the request is denied, and the agent must wait for the limit to refill. Crucially, rate limits can be scaled based on context, such as the namespace of the resources being accessed. Malhotra highlighted an example where a rate limit on deletions was implemented via a webhook, preventing the previous incident. The bypass flag in such systems should ideally prompt the agent to ask a human to perform the action directly.

Primitive 3: Tripwires Over Allow Lists

While allow lists are static guesses about an agent's needs, tripwires provide a mechanism to gather data on agent behavior after the fact. For cheaper actions, agents can operate freely, with their actions logged and monitored. Tripwires act as "smoke detectors," alerting humans when aggregate behavior deviates from the norm (e.g., an unusually high number of investigation threads launched by an agent).

Malhotra shared an example where a tripwire detected an excessive number of investigation threads, revealing an underlying infrastructure issue. The fix involved a simple prompt adjustment, guiding the agent to correlate failures before launching individual investigations. This iterative improvement is a key advantage of tripwires over static allow lists.

The Lens: The Undo Test

The "undo test" serves as a critical lens for evaluating the safety of agent actions. It poses two questions: Can the agent undo its own action, and if it gets it wrong, how bad is the impact? If an agent can self-correct and the potential impact is acceptable, the action can be logged and allowed. If either answer is no, a "second key" or human oversight is required.

Policy as Text and Infrastructure

Malhotra emphasized that policy for AI agents needs to exist in two layers: text and infrastructure.

  • Text: Prompts and context files shape the agent's intent and provide reasoning, but they are advisory and lack enforcement.
  • Infrastructure: This includes proxy layers and admission webhooks that enforce limits and budgets deterministically, bounding how wrong an agent can go.

He illustrated this with a diagram showing how each agent session has a proxy that stamps its identity onto every outbound call. This identity is then used by various safeguards throughout the system, ensuring accountability and control without the agent itself needing to manage its provenance.

Key Takeaways

Malhotra summarized his core message with four key points:

  • Asymmetric Verbs: Grant agents access to actions that "fail out loud" and keep humans involved for actions that "fail silently."
  • Rate Limits: Implement ceilings on every write operation that can refill autonomously, eliminating the need for ticket filings.
  • Tripwires: Monitor aggregate behavior rather than individual calls and use simple text-based fixes to guide agent actions.
  • Undo Test: Use this as a sizing tool for all primitives, asking if the agent can recover from its actions and what the impact of failure would be.

Underpinning all these principles is the idea that identity must come from the infrastructure, not the request itself. Getting this foundational rule right, Malhotra concluded, simplifies the process of safely integrating AI agents into production workflows.

Mentioned in this story

TopicsAnthropicAI Engineer World's FairAI AgentsCIKubernetesCloud Computing

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.