Turbopuffer's Approach to Daily Database Upgrades Across 100+ Clusters

Operating Diverse Database Deployments

Turbopuffer manages over 100 database clusters, a number that has doubled in the last six months, supporting public SaaS, single-tenant SaaS, and Bring Your Own Cloud (BYOC) models. This growth necessitates a robust system for frequent upgrades, often multiple times a day, to introduce new features and improvements.

Challenges with BYOC Environments

A significant challenge arises with BYOC clusters, which reside within customer cloud accounts where Turbopuffer typically lacks direct access credentials. Unlike some vendors who request standing admin credentials, Turbopuffer avoids this to prevent security, compliance, and billing overhead for customers. This constraint means operations must be designed to function without direct SSH or kubectl access.

Kubernetes-Native Operational Model

To address the operational challenges across diverse environments, especially BYOC, Turbopuffer employs a Kubernetes-native solution. Each cluster runs a local agent that implements a state machine. Operations, such as upgrades or tidy-ups, are expressed as a single Kubernetes Custom Resource Definition (CRD) called TurbopufferOperation.

A Kubernetes controller drives each custom resource through a reconciliation loop until it reaches a terminal state. This design allows operations to advance autonomously, even if connectivity to a central control plane is lost. BYOC customers can also configure operations to require approval or adhere to maintenance windows, which are integrated as waiting states within the CRD.

Autonomous and Durable Operations

The core principle is to define generic yet finite states within the CRD that can model all current and future operations. This ensures the reconciler can always guide operations to completion without manual intervention. The operational state is stored durably in the cluster's etcd, providing resilience against agent crashes or connection losses.