Cloudflare launches K2 in public beta, a serverless event streaming service

Introduction of Cloudflare K2

Cloudflare has launched K2 in public beta, a new serverless event streaming primitive on its Developer Platform. K2 addresses the challenge of traditional RPC architectures where producers and consumers must align in scale and time, often leading to dropped events if consumers cannot keep up or become unavailable.

Decoupling Producers and Consumers

K2 functions by decoupling data producers and consumers, inserting a service that absorbs writes while allowing independent readers to consume data at their own pace. This design ensures that events are stored durably as an ordered log, preventing data loss even during consumer downtime and supporting multiple consumers with independent processing needs.

Technical Foundation and Scaling

Under the hood, K2 implements a partitioned, durable log utilizing Cloudflare's R2 object storage. This architecture enables K2 to scale to vast quantities of data and support long-term retention. It is designed to operate across Cloudflare's global edge network, which spans over 335 cities, addressing the unique challenges of running stateful services on such distributed infrastructure.

Origin and Purpose

Cloudflare initially developed K2 to serve as a durable buffer on the edge, specifically as the ingestion layer for Basin Pipelines. Pipelines, which operates on a pull-based model, required a system to store events durably before they are processed, transformed, and written to R2, ensuring no events are lost once accepted into the stream.

Addressing Edge Infrastructure Challenges

The development of K2 was necessary because traditional distributed systems software like Apache Kafka are not easily deployable across Cloudflare's extensive edge infrastructure. Cloudflare's unique architecture necessitates rethinking how such systems are built and operated to handle the global scale and distributed nature of its network.