Cloudflare Makes AI Agents Smarter with RFC 9457

AI agents are no longer theoretical; they are integral to production infrastructure, making billions of daily HTTP requests. Previously, when these agents encountered errors, they were met with the same verbose, human-designed HTML pages meant for browsers. This created a significant inefficiency, providing agents with clues rather than clear instructions and wasting valuable tokens.

Addressing this gap, Cloudflare has begun returning RFC 9457-compliant structured error responses in both Markdown and JSON formats. This replaces the cumbersome HTML with machine-readable instructions, offering agents actionable guidance instead of obstacles.

From Clues to Instructions

Instead of a generic 'You were blocked' message, an AI agent will now receive specific instructions like 'You were rate-limited — wait 30 seconds and retry with exponential backoff.' Similarly, an 'Access denied' error will be replaced by 'This block is intentional: do not retry, contact the site owner.'

These structured responses dramatically improve efficiency, cutting payload size and token usage by over 98% compared to traditional HTML error pages. This saving compounds quickly for agents navigating complex workflows that encounter multiple errors.

Machine-Readable Errors by Default

Cloudflare's new system automatically delivers RFC 9457-compliant structured responses for all 1xxx-class errors, which cover platform-level issues like DNS resolution failures, access denials, and rate limits. This applies network-wide without any configuration needed from site owners, while browsers continue to receive the familiar HTML experience.

The structured responses include YAML frontmatter for machine-readable fields and prose sections for explicit guidance. For JSON responses, these fields are presented as a flat object. Key fields like error_code, retryable, and owner_action_required enable agents to classify failures, implement backoff logic, and determine when to escalate.

The JSON responses adhere to the RFC 9457 standard for HTTP API error reporting, ensuring compatibility with standard HTTP clients. Cloudflare extends this standard with operational fields such as error_category, retry_after, and ray_id, providing deterministic data for logging and support.

Size Reduction and Token Efficiency

The impact on efficiency is substantial. A comparison for a 'rate-limit' error (1015) shows a Markdown response uses 58.5x less data and 64.5x fewer tokens than its HTML counterpart. JSON responses offer similar gains, with an 48.1x reduction in size and 55.7x fewer tokens.

This reduction in size directly translates to lower token costs for AI agents, which is critical for managing model spend and improving recovery loops when multiple errors occur.

Clear Actions for Every Error

Each 1xxx error is mapped to a specific error_category, transforming error handling into straightforward routing logic. Categories range from access_denied and rate_limit to dns and config errors, each with clear directives for agent action.

The retryable and owner_action_required fields provide explicit control flow signals. This allows agents to move beyond brittle heuristics, like simply retrying on a 429 status code, to implementing precise logic based on stable, structured data.

Agents can now parse these structured fields once and branch on stable data. A common pattern involves checking if retryable is true, waiting for retry_after seconds, and retrying. If owner_action_required is true, the agent escalates; otherwise, it fails fast.

How to Utilize Structured Errors

To receive these structured responses, AI agents should send the Accept header with values like text/markdown, application/json, or application/problem+json. For testing, specific error codes can be accessed via the /cdn-cgi/error/ endpoint on any Cloudflare-proxied domain.

Cloudflare's implementation ensures that the first explicitly requested structured type takes precedence, providing a predictable response format. This shift fundamentally changes how AI agents interact with web infrastructure, moving from inference based on human-readable pages to direct execution of machine-defined policies.