How we built Cloudflare's data platform and an AI agent on top of it
Cloudflare built a unified data analytics platform called Town Lake to streamline access to its vast amounts of data, spanning over 100+ countries. This platform provides a single SQL interface to all of Cloudflare's data, ensuring consistency and accuracy in querying. Town Lake is built on R2 storage, Workers for compute, and Cloudflare Access for authentication, with a focus on security, governance, and scalability.
Town Lake's architecture is a data lakehouse, combining query engines, metadata layers, and data cataloging to deliver fast and secure data access. Its components include a query engine powered by Apache Trino, a managed Apache Iceberg service for storage, a metadata catalog for data lineage and ownership, and an access control service for secure authentication.
Built on top of Town Lake is Skipper, an AI data agent that runs on plain English queries to provide correct, auditable answers in seconds. Skipper aims to empower anyone at Cloudflare to access and analyze the stream of data flowing through their network
NetworkingSecurity