AI Agents Learn Human Web Browsing via CDP

Corey Gallon, a speaker at the AI Engineer World's Fair, discussed the "dark arts of web automation," revealing how AI agents can be trained to interact with websites in a human-like manner. Gallon's presentation, titled "The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans," explored techniques for overcoming website defenses that often block automated bots.

The Core Thesis: A "Meatbag with a Mouse"

Gallon's central premise is that a CDP (Chrome DevTools Protocol) browser is indistinguishable from a human user to websites. "If you have a browser, an agent drive a browser using the Chrome DevTools protocol, your agents clicks and keystrokes travel the exact same path inside Chrome that yours do," he explained. This allows AI agents to perform actions like booking, sending emails, and filling forms without human intervention.

The Three Pillars of Human-like Browsing

To achieve this human-like web interaction, Gallon outlined three essential components:

• CLI, not MCP: Agents should be driven by Command Line Interface (CLI) tools rather than Message Queue Telemetry Transport (MQTT) servers. StartupHub.ai data indicates that while both approaches achieve similar task success rates (around 83%), CLIs offer superior reuse, speed, and cost-efficiency. CLIs allow code to be written once and reused, whereas MCPs require model calls at every turn, leading to slower execution and higher token costs. Gallon cited Anthropic's report that CLIs can be up to 75 times cheaper.
• CDP: The Whole Browser: The Chrome DevTools Protocol, familiar to developers as the F12 developer console, provides the necessary interface for agents to interact with browsers. Gallon highlighted that while the CDP surface area is vast, a small subset is sufficient for human-like interaction, covering visual senses (DOM, accessibility tree, screenshots), auditory senses (network traffic, console logs), and operational actions (clicks, keystrokes, navigation).
• A Loop on a Ladder: This involves a cyclical process of "sense, act, verify." An agent senses the page, performs an action, and then verifies the outcome, potentially through a different sensory channel than the initial action. If the page resists, the agent must "climb the ladder" of techniques, escalating to more human-like interactions only when necessary.

The "Meatbag Ladder": From Fake to Human

Gallon introduced the "Meatbag Ladder" as a tiered approach to overcoming website resistance:

• Rung 1 (Fake): The simplest approach involves using synthetic JavaScript clicks. This is the default and most efficient method when it works, as seen in the Outlook example for sending personalized emails.
• Rung 2 (Real): When synthetic clicks fail, agents can use CDP's input domain to dispatch events that mimic real mouse clicks, making them indistinguishable from human input. This was demonstrated with adding items to a cart on an e-commerce site.
• Rung 3 (Human): For the most challenging defenses, agents need to simulate human behavior, including mouse paths with jitter, visual perception, and complex interaction patterns. This rung is necessary for defeating advanced CAPTCHAs like Cloudflare's Turnstile, MT Captcha, and reCAPTCHA v2.

Defeating CAPTCHAs: The Final Boss

Gallon detailed how his methodology can overcome sophisticated CAPTCHAs. For Cloudflare's Turnstile, which uses isolated boundaries and shadow roots, the solution involved calculating the checkbox's position and firing a trusted click directly at it. For image-based CAPTCHAs like MT Captcha, agents use vision capabilities to identify characters in noisy images and then input the answer using trusted keystrokes. The most challenging, like reCAPTCHA v2, requires a hybrid approach: deterministic code for the logic and an AI agent for the visual recognition task, ensuring speed and accuracy.

Explore, Then Automate

The process involves two phases: first, manually running the loop and climbing the ladder until the task is solved, and second, automating the solution by writing it down as code or an agent skill. This ensures the solution is repeatable and durable.

OpenAI's Ban Threat and Resolution

Gallon humorously recounted how OpenAI threatened to ban his account for his work on web automation, citing "cyber abuse with a web browser." After a discussion, OpenAI rescinded the threat, restoring his access. He encouraged the audience to explore the "Chrome Agent" tool he developed, which is available in the Python ecosystem.