Netflix Enhances Data Movement Architecture for Cassandra to Iceberg

Background on Data Movement at Netflix

Netflix has developed a unified platform called Data Bridge to manage batch Data Movement. Previously, various custom connectors were created by different engineering teams for their specific needs, but these have now been centralized into a cohesive system with a catalog of connectors and a user-friendly interface.

Cassandra to Iceberg Data Movement

Apache Cassandra is essential for critical Netflix applications like Member, Billing, and Recommendations. The transition of data from Cassandra to Apache Iceberg is vital for analytics and operations, handled by a custom-built connector named Casspactor.

Casspactor managed approximately 1,200 data movements daily, transferring roughly 3 PB of data. Its reliance on a complex architecture for metadata dependencies highlighted its fragility.

Challenges with the Legacy System

The Casspactor system faced challenges, particularly related to its dependency on multiple independent systems to ascertain the completeness and content of backups. This complexity led to inefficiencies and limitations in processing capability, prompting a need for a reassessment of the data movement architecture.

Transition to a New Architecture

The new architecture aims to simplify the metadata handling and improve overall resilience in the data movement process. By utilizing existing backup infrastructures more effectively, it reduces the operational burden and enhances the efficiency of data transfers to Iceberg tables.