Netflix Conductor Workflow Engine Rebuilt for Scale After Open-Source Discontinuation

Conductor's Evolution Post-Open Source

Netflix previously discontinued maintenance of the open-source Conductor project to concentrate resources on its internal fork. This internal version has since undergone a complete rebuild to address significant growth in usage and demand.

The internal Conductor engine, which orchestrates microservices into business processes, has seen its workflow executions increase dramatically. It now clears a billion executions roughly every quarter, a substantial rise from previous milestones.

Architectural Changes for Scale

The increased scale necessitated a rethinking of nearly every layer of the Conductor system. The data plane was modernized, and the evaluation engine was rewritten. Changes were also implemented in how Conductor allocates work, controls concurrency, and manages compute resources.

These architectural changes were driven by the need to sustain the growing volume of workflows and tasks, impacting the core evaluation engine and underlying layers.

Expanded Internal Usage

Conductor's internal footprint has expanded significantly. It now supports approximately 200,000 workflow definitions, owned by around 150 applications across various Netflix departments, including Content and Studio Engineering, Ads, and Games.

The runtime side has also grown, with the system executing about 420 million workflows monthly and 5 to 20 million daily. Teams are adding hundreds of new workflow types each month.