Alex Shaw: "Everything Is a Rollout" in AI Agent Evaluation

Alex Shaw from the Lode Institute presented at the AI Engineer World's Fair, delivering a talk titled "Everything Is a Rollout." The presentation focused on the Harbor framework, an agent evaluation and reinforcement learning environment framework, drawing parallels between traditional software engineering and the emerging field of agent development.

The Evolution of Software Engineering and the Rise of Agents

Shaw began by taking the audience back to 2018, a year marked by cultural milestones like the release of Avengers Infinity War and the rebranding of Musical.ly to TikTok. He contrasted the software engineering practices of that era, exemplified by a humorous tweet about reviewing a 400-line pull request, with the current shift towards agentic coding. He quoted Francois Chalet, who stated that "Agentic coding is a form of machine learning. Generated code is best treated as a blackbox artifact whose behavior and generalization should be managed via empirical evaluation like with any ML model." Shaw extended this idea, claiming that "agent performance itself is best treated as a blackbox artifact."

This fundamental shift, Shaw explained, means that agent development is more akin to machine learning than traditional software engineering. He illustrated this by comparing key machine learning concepts with their agent development counterparts:

• Training data maps to environments.
• Test and validation sets map to evaluations (also environments).
• Model weights map to skills, prompts, tools, and the model itself.
• Loss functions map to environment rewards and feedback.
• Backpropagation/optimizers map to context-based optimization algorithms or running agents in loops.
• Gradient descent steps map to pull requests.
• Overfitting maps to reward hacking or overfitting within agent development.

Harbor: A Framework for Agent Evaluation

Shaw introduced Harbor as a solution to the need for new tools in agent development. Harbor is described as:

• A format for specifying agentic environments.
• An open-source framework for performing rollouts in parallel, supporting any agent, model, sandbox, or task.
• A registry of popular training and evaluation environment sets, with hundreds already available.

The core concept of evaluating an agent, Shaw explained, relies on environments. An environment requires an instruction, a place for the agent to act (a sandbox), and a verifier to assess the outcome. Harbor standardizes environment specification with a file directory layout, promoting interoperability.

The Rollout Process and Use Cases

The Harbor rollout process involves passing a sandbox to an agent, which then runs until a stopping condition is met, producing a trajectory. This trajectory is then passed to a verifier, which generates rewards. These rewards are aggregated across multiple rollouts to form the evaluation dataset.

Shaw highlighted several use cases for the Harbor rollout process:

• Evaluating Agent Product Building: Companies can use Harbor to test which coding agents or models perform best on their internal codebases and use cases.
• Evaluating Agent Product Usage: Companies can assess how well agents interact with their products, particularly in headless modes, to improve product usability for AI.
• Evaluating Agent-Powered Features: Assessing how agents enhance product features.
• Evaluating Internal Process Automation: Automating internal company processes with agents.

He also demonstrated the command-line interface for Harbor, showcasing its ability to run benchmarks with multiple agents and models in parallel, emphasizing the importance of parallelization for tightening development loops and maximizing throughput.

The Ubiquity of Evals and Future Outlook

Shaw asserted that "every single company that uses computers" needs evaluations, as AI can automate processes and increase productivity. He quoted Satya Nadella, who stated, "If you want to build an agentic system, start with the eval that matters... and your ability to grade that outcome. And then say, I welcome all models." Shaw echoed this sentiment, noting that having a robust evaluation system empowers companies to consider any model without relying on third-party assurances.

The presentation concluded with a list of notable projects and integrations utilizing Harbor, including FrontierSuite, BankerToolBench, RuneBench, Scale Atlas, AutoAgent, AfterQuery, and LangChain, underscoring Harbor's growing significance as a common language for agent evaluation and development.