Paul Klein: AI Agents Need Better Web 'Harnesses'

Paul Klein, founder of Browserbase, delivered a compelling talk at the AI Engineer World's Fair, focusing on the challenges and opportunities in bringing AI agents onto the World Wide Web. Addressing a room of presumably sleep-deprived attendees, Klein opened by acknowledging the current state of AI agents, noting that despite significant advancements in model capabilities, agents still struggle to effectively interact with the web.

The Web's Human-Centric Design

Klein began by highlighting a fundamental issue: "The web wasn't built for agents. It was built for people." This inherent design, he explained, creates significant hurdles for systems attempting to automate web interactions. He drew on his own early career experiences in web automation, describing the pain of maintaining scripts that frequently broke due to website changes, inefficient data representation, and various blockers.

Beyond Model Capabilities: The Harness Problem

While acknowledging the rapid progress in LLM capabilities, particularly in long-context tasks and interface utilization, Klein argued that the primary limitation for web agents is no longer the models themselves. Instead, he stated, "agents are missing the right harness and tools." He referenced Andrej Karpathy's observation that the systems and scaffolding around an LLM, its 'harness', are crucial for effective real-world interaction. Klein emphasized that investing in harness engineering can unlock significant potential from even existing models.

Harness Engineering in Action

Klein pointed to the success of harness engineering in the coding domain, citing examples like Cursor and Claude Code, which achieved superior results by optimizing their harnesses for coding tasks. He asserted that this principle applies equally to computer use and web agents, and that companies can achieve better model performance by building domain-specific harnesses, rather than waiting for models to inherently improve.

The Massive Non-Coding Opportunity

Klein stressed the significant untapped potential in non-coding agentic use cases. He presented data suggesting that while coding and software engineering tasks see around 60% solve rates, computer-use and web automation tasks lag at approximately 16%. This gap represents a massive opportunity, particularly for businesses operating on older web technologies. He argued that solving these computer use challenges is key to accelerating AI's diffusion into the broader economy.

Key Pillars for Reliable Browser Agents

Klein outlined three essential components for building effective browser agents:

• Multi-modal: Agents should not be limited to a single model type. Combining coding and computer-use models, or utilizing different models for different task complexities, can lead to better performance. He noted that many production-ready agents write code alongside browser interactions for efficiency.
• Harness Engineering: Beyond basic scripting, harnesses should incorporate features like memory and skills. He highlighted Browserbase's browser.sh, which publishes website-specific skills, allowing agents to perform tasks more efficiently by leveraging existing knowledge. Optimized token usage and context presentation are crucial here.
• Infrastructure: Reliable and consistent infrastructure is paramount. Klein criticized the common practice of using personal machines like Mac Minis for running agents, emphasizing the need for scalable, SOC 2 compliant solutions that ensure consistent page rendering and inputs/outputs.

Improving the Web Itself

Beyond engineering agents, Klein underscored the need to improve the web to be more agent-friendly. He identified three key areas:

• Accessibility: Leveraging standard accessibility trees and ARIA tags provides agents with structured information for navigation and interaction. Features like WebMCP in Chrome allow agents to issue tool calls to websites directly.
• Authentication: Securely handling agent authentication is critical. While current methods like password sharing or service accounts have limitations, solutions like WorkOS's OffMD are emerging to help agents sign up and manage their own accounts.
• Trust: Differentiating between good and bad bots is essential. While CAPTCHAs are common, they are not always effective. Frameworks for agent identity and verification, similar to digital certificates, are needed to build trust.

Browserbase Agents: A Battery-Included Solution

Klein concluded by introducing Browserbase Agents, a new product designed to provide a comprehensive agent and harness solution for automating web tasks. He stated that the goal is to eliminate the need for developers to reinvent the wheel, allowing them to focus on solving customer problems rather than rebuilding core browsing agent infrastructure. The platform aims to offer scalability, model agnosticism, agent identity solutions, and observability, enabling agents to continuously improve through data feedback loops.

Klein expressed optimism for the future, predicting that the current challenges in agent development will be overcome as models, techniques, and tools continue to improve, leading to a surge in adoption and innovation in the coming year.