Yelp successfully migrated its Cassandra fleet from legacy self-managed Kubernetes clusters to Amazon EKS. This migration covered approximately 90 production clusters, including a 54-pod user-data store, which took about 12 hours to complete. The process resulted in zero downtime for customers and no data loss.
Previously, Yelp had moved stateless workloads to EKS, but Cassandra remained due to Kubernetes' lack of built-in StatefulSet migration across clusters. Traditional methods, such as adding new nodes and shutting down old ones, would have temporarily doubled compute capacity and incurred significant data transfer costs, potentially taking weeks for large clusters. Yelp also considered regional failover and rebuilding or creating a new Cassandra datacenter in EKS, but these options were deemed less transparent to customers.
Yelp's solution, termed "Rolling PV Transplant," leveraged the fact that an EBS volume is bound to an Availability Zone, not a Kubernetes cluster, and a PersistentVolume acts as a pointer to a volume ID. The team pre-created PersistentVolumes and PersistentVolumeClaims in EKS referencing existing volume IDs. An identical StatefulSet was then created in EKS, with pods waiting in a ContainerCreating state. Automation deleted one pod at a time from the legacy cluster, waited for the volume to detach, and allowed the EKS pod to claim it. The pod rejoined the Cassandra ring with its original data and token range, and the process repeated for subsequent pods. Cassandra's replication mechanism tolerated the brief offline period for each node.
During the migration, a skipReconciliation: true flag was set on custom resources in both clusters to prevent conflicts between two operators managing the same logical ring. This flag was cleared on the EKS side post-migration. The method allowed approximately 1,000 TiB of data across 1,000 pods to be moved by pointer rather than over the network. The rollback process was the reverse of the migration, which Yelp utilized a few times for issues like a missing subnet definition or certificate mismatch, restoring pods within minutes. The production rollout was executed by one engineer over three weeks, migrating three to six clusters daily.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
Yelp migrated its Cassandra fleet from self-managed Kubernetes to Amazon EKS by re-pointing existing EBS volumes to new pods, avoiding data streaming. This method allowed the migration of approximately 90 production clusters with zero downtime and no data loss. The approach circumvented the limitations of Kubernetes StatefulSet migration across clusters and reduced data transfer costs.