Cloudflare's AI Data Agent

Cloudflare, processing over a billion events per second across its global network, faced a significant data challenge. Information was scattered across dozens of databases, cloud buckets, and streaming platforms, making even simple queries a complex, knowledge-intensive task. This data sprawl hindered effective insight generation.

To combat this, Cloudflare developed two internal tools: "Town Lake," a unified data analytics platform, and "Skipper," an AI data agent built on top of it. Town Lake provides a single SQL interface to all of Cloudflare's data, while Skipper enables employees to ask questions in plain English and receive auditable answers rapidly.

The Data Sprawl Problem

Hyper-growth often leads to data fragmentation. Cloudflare experienced this with too many disparate systems, sampled data unsuitable for critical functions like billing, reliance on external vendors for internal reporting, and a lack of discoverability for data assets.

This situation fostered a culture where data infrastructure was viewed as a secondary function rather than critical technology.

The Vision for Unified Data

The goal was to create a centralized, secure platform where authorized users could access fresh, accurate data. This included handling both high-volume sampled data for dashboards and precise, unsampled data for billing or security investigations.

Key requirements included automated PII detection, robust security and governance, auditable access logs, and time-bound permissions. Crucially, the entire system was to be built using Cloudflare's own product suite, such as R2 for storage and Workers for compute.

Ultimately, the vision was an interface that democratized data access, moving beyond SQL to empower anyone with a need to know.

Town Lake: The Data Platform

At its core, Town Lake employs a data lakehouse architecture. This combines a query engine with object storage and a metadata layer to present data as a unified database.

Key components include:

• Query Engine: Apache Trino is used to query data across various sources, including Postgres, ClickHouse, and Iceberg tables on R2, without materializing intermediate results.
• Data Catalog: Cloudflare's managed Apache Iceberg service, R2 Data Catalog, stores data with features like schema evolution and time travel, optimizing storage costs based on data recency.
• Metadata Catalog: DataHub centralizes metadata, including table schemas, ownership, lineage, and glossary terms, aiding data discovery.
• Access Control: Lifeguard manages access rules, integrating with Cloudflare Access for authentication and providing dynamic JSON policies to the query engine.
• PII Detection: Skimmer, a PII scanning service, uses Workers AI to classify columns for sensitive data, flagging findings for review.
• Transformation Engine: Transformer, built on Workflows, orchestrates ELT processes using SQL transformations defined in YAML.
• Ingestion: A dedicated orchestrator manages the extraction, transformation, and loading of data from operational systems into R2 as Iceberg tables.

Governance by Construction: Default-Closed

Town Lake adopts a default-closed security model. Tables are inaccessible until reviewed and approved, with automated scanning for PII.

This process is streamlined through self-serve workflows, where users can easily request reviews for unapproved tables. Sensitive columns are hidden by default, with PII access granted per session and logged.

Skipper: The AI Data Agent

Skipper acts as a conversational AI data agent, translating natural language questions into validated data queries. It leverages Town Lake's capabilities and Cloudflare's developer platform, including Workers AI.

Users interact via a chat interface, posing questions like "Show me the top 10 customers by R2 storage cost." Skipper handles table discovery, query generation, execution, and result presentation, including charts and dashboards.

The agent supports iterative refinement of queries and includes closed-loop reasoning to investigate and correct potential errors.

Context is King for LLMs

To mitigate LLM hallucinations and ensure accurate results, Skipper employs multiple layers of grounded context:

• Schema and Usage Metadata: DataHub provides comprehensive schema information and historical query patterns.
• Human Annotations: Descriptions and tags in DataHub offer human-curated context.
• Code-Derived Knowledge: SQL transformation logic from the Transformer pipeline enriches understanding of data meaning.
• Curated Data Models: Human-written documents guide users on how to interpret key data concepts.
• Runtime Introspection: Live queries to Trino serve as a final safety net for context verification.

Skipper's tools are accessible through Workers AI and an MCP server, offering flexibility for different user workflows.