Cloudflare D1 Scales to Millions of Databases, Supporting 65x Growth in Two Years

D1 Control Plane Scales for Millions of Databases

Cloudflare's D1, a managed serverless database built on SQLite, has undergone architectural changes to scale its control plane. These changes enable D1 to support platforms managing millions of databases, with hundreds of thousands active concurrently. This addresses a growing trend in application development where platforms utilize many small databases, often one per user, application, or tenant, rather than a few large clusters.

Significant Growth in D1 Usage

D1 has experienced substantial growth in its user base. Over the past two years, the number of live, non-deleted databases on D1 has increased by 65 times. In the last year alone, there was a 15x growth in live databases and an 18x growth in daily active databases. An active database is defined as one that executes at least one query within a day.

Addressing New Database Architecture Trends

Traditional database architectures typically involve one or two large database clusters per application. However, AI-assisted development and vibe coding platforms are shifting this model towards applications requiring numerous smaller databases. D1's architecture is designed to accommodate this shift, allowing users to create millions of databases and pay only for their usage, without performance bottlenecks from centralized control planes.

Impact on Cloudflare Developer Platform

The architectural patterns developed for D1's control plane are applicable to other services on Cloudflare’s Developer Platform. This scaling effort ensures that D1 can support large-scale platforms, such as OpenAI's ChatGPT Sites, which uses D1 for features like custom metrics, leaderboards, and comments. Load testing was conducted periodically during the changes to guide optimization efforts.