The internet's next evolution is interacting with AI agents, a shift that requires websites to adapt beyond human and search engine protocols. Cloudflare is launching a new tool, isitagentready.com, to help site owners gauge and improve their compatibility with these emerging AI systems.
This initiative also includes a new dataset on Cloudflare Radar tracking the widespread adoption of agent standards across the web.
The Web's Agent Readiness Gap
The short answer to whether the web is ready for AI agents is a resounding no. Cloudflare Radar analyzed 200,000 top websites, excluding categories irrelevant to AI interaction, to measure adherence to agent standards.
Key findings reveal significant gaps: while 78% of sites have a robots.txt file, most are configured for traditional crawlers, not AI. Only 4% declare AI usage preferences, and even fewer support emerging standards like Markdown content negotiation (3.9%) or MCP Server Cards and API Catalogs (fewer than 15 sites).
This early stage presents a substantial opportunity for early adopters.
The daily brief
Get the free daily brief with the day's AI news, new AI startups and funding rounds every morning.
Join 64,000 other subscribers.
Understanding the Agent Readiness Score
The isitagentready.com tool provides a score based on four core dimensions:
- Discoverability: How easily agents can find your site and its content (e.g., robots.txt, sitemaps, Link Headers).
- Content: The format and accessibility of your content for AI (e.g., Markdown, llms.txt).
- Bot Access Control: Rules governing agent access and usage (e.g., Content Signals, AI bot rules).
- Capabilities: How well your site supports agent actions and integrations (e.g., Agent Skills, API Catalogs, OAuth).
The tool also checks for agentic commerce standards, though these do not currently impact the score.
Driving Adoption Through Tools
Similar to Google Lighthouse for web performance, the Agent Readiness score aims to guide site owners toward best practices. Failing checks provide actionable prompts that can be given to coding agents for implementation.
Cloudflare's own Developer Documentation has been overhauled as a case study, adopting these standards to improve AI tool performance and cost-efficiency.
Integrating Agent Readiness
Cloudflare's URL Scanner now includes an 'Agent Readiness' tab, offering detailed reports and actionable guidance alongside existing site analysis data. This capability is also available programmatically via the URL Scanner API.
