Netflix implements tiered storage for TimeSeries data, moving older data to S3

Introduction to TimeSeries Tiered Storage

Netflix introduced tiered storage solutions for its TimeSeries Abstraction platform, which stores and queries temporal event data. The platform has grown to manage multiple petabytes of data, making cost-efficient storage a priority.

The new approach focuses on moving older, less frequently accessed data from expensive hot storage to cheaper object storage, specifically Amazon S3. This strategy aims to reduce infrastructure costs while ensuring data remains accessible.

Two Solutions for Cold Data Management

Netflix developed two distinct solutions for managing cold TimeSeries data. The first solution utilizes Apache Kafka and Flink to stream data directly into S3. This method provides a continuous pipeline for data offloading.

The second solution leverages existing Cassandra backups in S3 and integrates pluggable Memtables to enable direct reading of data from S3. This evolution allows for native Cassandra-based cold tier access.

Addressing Data Growth Challenges

The need for tiered storage arose from several challenges associated with the expanding TimeSeries datasets. These include increasing data retention demands, escalating costs of maintaining large datasets in hot storage, and the sparse access patterns for historical data.

Maintaining dense clusters for infrequently accessed historical data proved prohibitively expensive. The new solutions address these issues by optimizing storage based on data access frequency, aiming to meet user and operational needs while reducing infrastructure spend.

TimeSeries Abstraction Overview

The TimeSeries Abstraction partitions datasets into multiple tables, each covering a specific time window called Time Slices. This partitioning allows for optimizations, such as making a table immutable once it crosses a write boundary.

This immutability enables further optimizations, including the safe migration of older data to less expensive storage tiers without affecting ongoing write operations.