Terminal-Bench 2.0 and Harbor Reset the Bar for AI Agent Evaluation

The recent launch party for Terminal-Bench 2.0 and Harbor, hosted by Mike Merrill and Alex Shaw, unveiled a pivotal shift in how AI agents are evaluated, moving firmly towards command-line interface (CLI) interactions as the gold standard for performance. This event, which included a fireside chat with industry leaders Andy Konwinski and Ludwig Schmidt, highlighted the critical need for robust, standardized benchmarks and tools to accurately measure the capabilities of increasingly sophisticated AI agents.

Mike Merrill, co-creator of Terminal-Bench, explained that the initial iteration, Terminal-Bench 1.0, was built on the premise that efficient agent-computer interaction would predominantly occur via CLI, rather than graphical user interfaces (GUIs). He illustrated this with a compelling example: "If you've ever tried to launch an EC2 instance from the GUI, it's a ton of menus, takes you 20 or 30 clicks. In comparison, there's just a single CLI command that can do this for you." This foundational belief in CLI's superior efficiency for expert tasks became the bedrock for their benchmarking efforts.

Terminal-Bench 1.0, launched in May, rapidly gained traction, with thousands of GitHub stars and Discord members, and adoption by all frontier AI labs. Its unexpected uses, from prompt optimization to CI/CD for agent deployments and even reinforcement learning, showcased its utility. Merrill noted, "Didn't see any of this coming," underscoring the community's hunger for agent evaluation tools. Yet, this initial success also revealed significant limitations. Some tasks were too easy, like a "hello-world" text file edit, allowing even weak agents to score. Others lacked reproducibility, such as a YouTube download task that fell victim to YouTube's anti-bot measures, rendering solutions ephemeral. Furthermore, tasks like playing Zork, while interesting, lacked inherent real-world value for evaluating practical agent performance.

Terminal-Bench 2.0 addresses these shortcomings with a comprehensive overhaul. The latest version comprises 89 tasks, many new or significantly improved, all meticulously based on real-world work. This iteration places an unprecedented emphasis on verification. Terminal-Bench 2.0 boasts 89 meticulously crafted tasks, each undergoing over 300 hours of human and LM-assisted verification. This rigorous process ensures tasks are solvable, realistic, and precisely specified, directly addressing the shortcomings of its predecessor. This dedication ensures a benchmark with the "highest possible quality standards," as Merrill proudly stated. New tasks include designing DNA primers for biologists, installing legacy operating systems in emulators, and bypassing cybersecurity filters, pushing the boundaries of agent capabilities in relevant, complex scenarios.

Central to this new approach is Harbor, a new package designed to standardize and streamline the agent evaluation and optimization workflow. Alex Shaw articulated a recurring problem faced by agent developers: the continuous cycle of building custom evaluation frameworks. Developers typically start with an agent, want to know if it's good, and then proceed to build an evaluation. This involves defining instructions, setting up a containerized environment, and running tests. The next hurdle is scale; running thousands of evaluations quickly leads to resource limitations. "But I'm running out of cores," Shaw quipped, highlighting the common bottleneck. The solution, he explained, is to run evaluations in the cloud, but even then, the core process remains repetitive.

Related Reading

• Google Cloud's Vertex AI Agent Engine Bridges the Production Gap for AI Agents
• Box CEO: AI Agents Redefine Enterprise Productivity
• Model Context Protocol: Streamlining AI Agent Interaction with Cloud Tools

This realization led to the development of Harbor. Shaw emphasized, "You should stop writing the same code over and over again." Harbor provides a standardized task format, pre-integrates popular agents and benchmarks, and offers out-of-the-box cloud deployments using platforms like Daytona, Modal, and EC2. In essence, Harbor abstracts away the infrastructural complexities of agent evaluation, allowing developers to focus on agent development itself. With just a few lines of code, users can execute thousands of sandboxed rollouts in the cloud for both evaluation and post-training. This dramatically simplifies the process of optimizing agent prompts through techniques like reinforcement learning, where agents learn by interacting with the environment, returning rewards and trajectories to update their internal weights.

Preliminary results from Terminal-Bench 2.0, showcased during the presentation, reveal frontier models like Codex and GPT-4 ranking highest, with their custom Terminal-2 harness demonstrating superior performance for many models. This indicates that the benchmark is effectively differentiating agent capabilities and pushing the state of the art. The launch of Terminal-Bench 2.0 and Harbor represents a significant collaborative achievement from the open-source community, providing essential infrastructure for the future of AI agent development.