Netflix Adopts Open-Source Apache Flink Autoscaler for 30,000+ Streaming Jobs

Netflix Shifts to Open-Source Flink Autoscaler

Netflix is transitioning its extensive streaming job infrastructure to the open-source Apache Flink Autoscaler. This move impacts more than 30,000 streaming jobs operating across multiple AWS regions. The decision follows an evaluation of their previous cluster-level autoscaling method, which proved less effective for complex, stateful data pipelines.

Addressing Limitations of Previous System

Since 2017, Netflix has utilized Apache Flink, developing its initial autoscaler around 2019. This system used cluster-level telemetry to adjust the total number of TaskManagers, achieving a 25% to 45% reduction in resource usage. However, its limitation was scaling at the cluster level, meaning all operators within a job shared the same scaling decision. This approach was not ideal for stateful pipelines with diverse processing needs across different operators.

Operator-Level Scaling and Cost Savings

The Apache Flink Autoscaler addresses this by estimating each operator's true processing rate from throughput and busy time, then calculating required parallelism for individual vertices. This method is detailed in FLIP-271, which focuses on autoscaling for heterogeneous streaming jobs. One Netflix team reported a 58% reduction in annualized Flink compute expenditure, resulting in approximately $1.1 million in annual savings.

Integration and Enhancements

Netflix integrated the new autoscaler with its internal control plane, rather than deploying it via the Flink Kubernetes Operator. This integration involved using a Spring Boot service with Temporal workflows to manage autoscaling decisions for individual jobs. Additionally, Netflix modified JobManager metric collection to support jobs with up to 3,000 subtasks, added server-side metric filtering, preserved forward connected subgraphs during scaling, and incorporated handling for sink backpressure.