Sunil Pai, Founder and Principal Systems Engineer at Cloudflare, recently took the stage at AI Engineer Europe to discuss the evolving capabilities of AI agents. His talk, titled "Code Mode: Let the code do the talking," explored the transition from traditional tool-calling mechanisms to a more sophisticated model where AI directly generates and executes code.
Understanding "Code Mode"
Pai began by highlighting the challenges of scaling AI agent interactions. "Tool calling gets weird at scale," he stated, explaining that while simple tools work well for short-run tasks, managing hundreds or thousands of tools and their associated API calls becomes inefficient and prone to errors. This complexity leads to slow response times and a breakdown in the model's ability to compose actions effectively.
The solution, as proposed by Pai, is to shift from explicit tool definitions to a "code mode." In this paradigm, the AI agent generates code, typically in a language like JavaScript or Python, that directly interacts with the underlying systems or APIs. This approach offers several advantages:
- Type safety: Code provides a more robust and type-safe interface compared to JSON definitions.
- Execution: The model can execute the generated code directly, streamlining the process.
- Flexibility: It allows for more complex logic, such as loops, conditional statements, and error handling, which are difficult to express with simple tool calls.
From 2,594 Endpoints to Two Tools
Pai shared a compelling example from Cloudflare's own experience. They reduced their API surface from 2,594 endpoints to just two core tools: a search tool and an execute tool. This simplification was achieved by enabling the AI agents to write code that interacts with these fundamental capabilities. The result was a dramatic reduction in tokens used per call (from 1,000 to approximately 1,000) and a significant increase in efficiency, with a 99.9% reduction in token usage per action.
"We were able to shrink that entire API surface and make it really fast," Pai explained. This transition allows the AI to interact with the system more natively, leveraging the full power of programming languages to express complex intents and actions.