Cloudflare’s AI Platform: an inference layer designed for agents

Here is a 2-3 sentence summary of the article: To efficiently run extra-large language models like Kimi K2.5 on Workers AI, Cloudflare employs a hybrid approach, disaggregating prefill (input token processing) and decode (output token generation) into separate inference servers, and leveraging load balancing and token-aware routing to optimize performance and latency. Additionally, Cloudflare optimizes for prompt caching using the x-session-affinity header, and incentivizes its use with discounted cached tokens, while also leveraging Mooncake Transfer Engine and Mooncake Store for efficient KV-cache sharing across multiple GPUs. These optimizations have resulted in significant improvements in tail latency and intertoken latency, with a 3x improvement in performance.

Cloudflare has announced a partnership with PlanetScale to provide developers with direct access to Postgres and MySQL databases for fast, full-stack applications using Cloudflare Workers. This integration allows developers to create PlanetScale databases from the Cloudflare dashboard and have them billed to their Cloudflare account. With the integration, databases can be automatically connected to Workers using Hyperdrive, a database connectivity service.