Cursor's Auto-review Balances Agent Autonomy

Agents need autonomy to be productive, but too much freedom can lead to risky, unintended actions, especially for local agents interacting with sensitive systems. Cursor's new Auto-review feature addresses this by treating agent autonomy more like a dial than a switch.

The core principle is simple: allow agents freedom when stakes are low, and apply caution when actions cross meaningful boundaries. This dynamic adjustment is managed by a specialized classifier agent that reviews actions in context before execution.

Judging Risk in Context

An agent's action is only as safe as its environment. The same command can be benign in one workflow and catastrophic in another. Understanding the relationship between the action, user intent, and potential consequences is key.

This realization drove the development of a classifier agent designed for nuanced judgment. The goal was a small, fast model capable of discerning if an action aligns with user intent, prioritizing leniency for low-risk scenarios and caution for high-risk ones.

Building the Classifier

The classifier must be both fast and accurate, operating directly within the agent's execution loop. Cursor leveraged its multi-model capabilities to test various models and reasoning modes, seeking an optimal balance.

An early finding was that simpler models weren't always faster; complex policy or tool calls could lead them to spend more time and tokens on inferior decisions. A small model with sufficient reasoning proved more effective.

To handle actions requiring environmental awareness, the classifier was made agentic. It can inspect the workspace using tools like `ReadFile` or `ListDir` when a command like `python script.py` could be safe or unsafe depending on the script's content.

Integrating the classifier directly into the parent agent's RPC stream, rather than a separate endpoint, minimizes latency, crucial for real-time decision-making.

Designing the Feedback Loop

When the classifier blocks an action, it doesn't immediately prompt the user. Instead, it returns an explanation to the parent agent. This allows the parent agent to often select a safer alternative without interrupting the user's flow.

This feedback loop's effectiveness hinges on user intent. The focus is not on whether an action appears risky in isolation, but whether it's justified by the user's request, enabling uninterrupted development for routine tasks while flagging high-consequence actions.

Testing the Classifier

Initial evaluations used internal developer session data to establish a baseline for normal agent behavior. This helped tune the classifier to catch risky actions without hindering routine development.

Synthetic data was also generated to cover rare but critical failure cases, such as agents attempting to read secrets or manipulate production data. Policy changes necessitated relabeling or regenerating evaluation sets to maintain accuracy.

Evals were run through the full backend loop, including tool use and classification, to test the complete process. Stability was assessed by checking for "flapping", cases where the classifier's decision varied inconsistently across multiple runs.

Minimizing Outright Blocks

Many agent actions are already covered by allowlists or sandboxing. The classifier primarily intervenes when contextual judgment is required.

Currently, Auto-review blocks about 4% of actions. Crucially, most blocks are handled by the parent agent, with only about 7% of total chats in Auto-review mode leading to a user interruption.

This contrasts sharply with some enterprise clients who previously saw around 40% of actions blocked. The system successfully prioritizes user experience by minimizing direct interruptions.

Refining Agent Autonomy

Auto-review is an evolving system, designed to adapt as agents become more capable. Initially focused on local agents in the desktop app, its principles are expected to guide autonomy governance across more platforms.

The aim is to grant agents meaningful autonomy while ensuring that decisions to slow down are context-dependent, not dictated by a single global setting. This approach enhances safety without reverting to a constant stream of approval prompts, allowing agents to continue working when safer alternatives exist.

StartupHub data

Cursor (Anysphere)

Cursor is an AI code editor built to make you extraordinarily productive by coding with AI.

Founded

2022

Location

San Francisco, United States

Valuation

$29.3B