Erik Hanchett: Cut AI Agent Token Costs

In the rapidly evolving world of AI agents, managing operational costs is becoming as critical as the agent's functionality. Erik Hanchett, a Developer Advocate at AWS, delivered a concise presentation titled "Your Agent Is Wasting Tokens and You Don't Know It," highlighting five key strategies to significantly reduce token expenditure and optimize AI agent performance.

Who Is Erik Hanchett?

Erik Hanchett is a Senior Developer Advocate at Amazon Web Services (AWS). In his role, he focuses on empowering developers to build and deploy applications, often by explaining complex cloud services and best practices. His expertise lies in making advanced technologies accessible and practical for development teams.

The Problem: Unseen Token Waste

Hanchett immediately dives into a common pitfall for developers building AI agents: the silent and often unnoticed overconsumption of tokens. This waste directly translates to higher operational costs and can impact the scalability and economic viability of AI-powered applications. He frames the presentation around five practical "fixes" to address this issue.

Fix 1: Cache the System Prompt

The first strategy addresses the repetitive sending of system prompts. Hanchett demonstrates a code snippet showing how to define an agent using AWS's Bedrock models. By setting a cache_prompt="default" parameter for the system prompt, the agent can store and reuse this information. This means the full, often lengthy, system prompt is only sent on the initial call. Subsequent interactions will use the cached version, saving tokens on every turn.

Fix 2: Route by Difficulty

Not all tasks require the most powerful and expensive language models. Hanchett suggests implementing a routing mechanism that dispatches tasks based on their complexity. His example shows a Python function pick_model(task) that checks if a task is simple. If it is, it returns a less expensive model like "claude-haiku." For more complex tasks, it defaults to a more capable, but costlier, model such as "claude-sonnet." This dynamic model selection ensures that developers are not overspending on simpler requests.

Fix 3: Offload Big Tool Results

When AI agents interact with tools, the output from these tools can be substantial, consuming many tokens. Hanchett highlights the inefficiency of sending the full, raw output of a tool back into the agent's context. For instance, a tool that retrieves a report might return 10,000 tokens worth of data. Instead, he advocates for offloading or summarizing these large results. The example shows a fetch_report function that retrieves data, stores it, and then uses a summarize function with a reference. Only the summarized output is then passed back to the agent, drastically reducing the token count sent in subsequent processing steps.

Fix 4: Cap Your Tool Loops

A critical issue that can lead to runaway token costs is when an agent gets stuck in an infinite loop of tool calls. Hanchett demonstrates how to prevent this by setting a max_iterations parameter on the agent. For example, setting max_iterations=8 will halt the agent's execution after eight consecutive tool calls, regardless of whether the task is complete. This acts as a safeguard against excessive token consumption from malfunctioning or looping agent behavior.

Fix 5: Trim the History

For conversational agents, maintaining a long history of interactions is crucial for context, but it can also lead to massive token usage. Hanchett introduces the concept of a "sliding window conversation manager." His example shows how to use a SlidingWindowConversationManager with a specified window_size=10. This approach means that only the last 10 turns of the conversation are kept in context and sent to the language model. Older messages are effectively dropped, preventing the context from growing indefinitely and significantly reducing token costs in long conversations.

Recap: Five Fixes for a Smaller Bill

Hanchett summarizes the five key strategies for optimizing AI agent token usage:

• Cache the System Prompt: Stop resending the static prompt every turn.
• Route by Difficulty: Use a cheaper model for simple calls.
• Offload Big Tool Results: Keep 10k-token blobs out of context by summarizing them.
• Cap Your Tool Loops: Implement max_iterations to prevent runaway calls.
• Trim the History: Use a window or summarize old turns to manage context length.

By implementing these practical measures, developers can ensure their AI agents are not only effective but also cost-efficient, making them more sustainable for production environments.