The recent launch party for Terminal-Bench 2.0 and Harbor, hosted by Mike Merrill and Alex Shaw, unveiled a pivotal shift in how AI agents are evaluated, moving firmly towards command-line interface (CLI) interactions as the gold standard for performance. This event, which included a fireside chat with industry leaders Andy Konwinski and Ludwig Schmidt, highlighted the critical need for robust, standardized benchmarks and tools to accurately measure the capabilities of increasingly sophisticated AI agents.
Mike Merrill, co-creator of Terminal-Bench, explained that the initial iteration, Terminal-Bench 1.0, was built on the premise that efficient agent-computer interaction would predominantly occur via CLI, rather than graphical user interfaces (GUIs). He illustrated this with a compelling example: "If you've ever tried to launch an EC2 instance from the GUI, it's a ton of menus, takes you 20 or 30 clicks. In comparison, there's just a single CLI command that can do this for you." This foundational belief in CLI's superior efficiency for expert tasks became the bedrock for their benchmarking efforts.
Terminal-Bench 1.0, launched in May, rapidly gained traction, with thousands of GitHub stars and Discord members, and adoption by all frontier AI labs. Its unexpected uses, from prompt optimization to CI/CD for agent deployments and even reinforcement learning, showcased its utility. Merrill noted, "Didn't see any of this coming," underscoring the community's hunger for agent evaluation tools. Yet, this initial success also revealed significant limitations. Some tasks were too easy, like a "hello-world" text file edit, allowing even weak agents to score. Others lacked reproducibility, such as a YouTube download task that fell victim to YouTube's anti-bot measures, rendering solutions ephemeral. Furthermore, tasks like playing Zork, while interesting, lacked inherent real-world value for evaluating practical agent performance.
Terminal-Bench 2.0 addresses these shortcomings with a comprehensive overhaul. The latest version comprises 89 tasks, many new or significantly improved, all meticulously based on real-world work. This iteration places an unprecedented emphasis on verification. Terminal-Bench 2.0 boasts 89 meticulously crafted tasks, each undergoing over 300 hours of human and LM-assisted verification. This rigorous process ensures tasks are solvable, realistic, and precisely specified, directly addressing the shortcomings of its predecessor. This dedication ensures a benchmark with the "highest possible quality standards," as Merrill proudly stated. New tasks include designing DNA primers for biologists, installing legacy operating systems in emulators, and bypassing cybersecurity filters, pushing the boundaries of agent capabilities in relevant, complex scenarios.
Central to this new approach is Harbor, a new package designed to standardize and streamline the agent evaluation and optimization workflow. Alex Shaw articulated a recurring problem faced by agent developers: the continuous cycle of building custom evaluation frameworks. Developers typically start with an agent, want to know if it's good, and then proceed to build an evaluation. This involves defining instructions, setting up a containerized environment, and running tests. The next hurdle is scale; running thousands of evaluations quickly leads to resource limitations. "But I'm running out of cores," Shaw quipped, highlighting the common bottleneck. The solution, he explained, is to run evaluations in the cloud, but even then, the core process remains repetitive.
