Netflix Transitions from In-House to Open-Source Flink Autoscaler

Dual Autoscaler Operation

Netflix currently runs two distinct autoscalers for its Apache Flink stream processing infrastructure. One is an in-house system developed years ago, and the other is an open-source solution from the Apache Flink community. The company is in the process of consolidating its operations onto the open-source autoscaler.

Motivation for Autoscaling

Netflix has utilized Apache Flink for stream processing since 2017, managing over 30,000 Flink jobs across multiple AWS regions as of 2026. These jobs, ranging from simple data shuttling to complex stateful pipelines, experience fluctuating loads. Provisioning resources for peak demand is inefficient, while provisioning for average loads leads to performance issues during surges. Autoscaling is critical to dynamically adjust resources without manual intervention.

The Initial In-House Solution

The first autoscaler, developed around 2019, operated as a separate stream-processing job on Mantis. It consumed real-time cluster metrics from Atlas, including CPU, network, Kafka lag, input-rate, and consume-rate signals. This system used a combination of lag-derived catch-up time, utilization thresholds, performance history, and regression analysis to make scaling decisions. Its design allowed it to function independently of the Flink platform.

Transition to Open Source

The open-source Flink autoscaler provides capabilities that the homegrown system lacked, particularly for more complex and diverse workloads. Netflix is gradually moving towards full adoption of the community-developed solution, aiming to streamline its autoscaling infrastructure and leverage the broader community's advancements. This shift reflects lessons learned regarding the costs and benefits of maintaining custom infrastructure versus adopting mature open-source alternatives.