Netflix Engineer Details Spark Experimentation Strategy for Production Pipelines

Challenge with Production-Scale Spark Jobs

Netflix's member-sessionizer, a critical data pipeline, processes tens of billions of hourly client events to consolidate user activity into viewing sessions. During an upgrade, the new job encountered critical failures, including Out-Of-Memory errors and severe shuffle spillage, which could not be replicated with sampled data.

These issues only manifested at full production scale due to data skew, where extreme long-tail sessions (e.g., an idle Netflix tab streaming overnight) generate significantly more telemetry events than typical sessions. Standard 1% data samples failed to capture these edge cases, making traditional debugging ineffective.

Developing a Concurrent Experimentation Strategy

To address the limitations of single-run testing, which took hours per attempt, a Netflix engineer shifted focus to running and monitoring multiple tests simultaneously. This involved launching five Spark jobs concurrently and tracking their results, even over unreliable connections like inflight Wi-Fi, while cloud servers handled the processing.

This strategy allowed for rapid iteration and identification of Spark bottlenecks and memory quirks that were only apparent under full production load. Each test run executed against a full hour of unsampled production data.

Leveraging Branch-Driven Isolation

A key component of this approach was utilizing Netflix's Dataflow tool, which supports branch-driven development. This feature allowed workflows, tables, and run namespaces to be tagged with specific branch names (e.g., pgn/cap-experiment).

This isolation mechanism, integrated with Maestro (Netflix’s workflow orchestrator), enabled the engineer to deploy and test changes in an isolated environment without impacting production, facilitating the concurrent execution of experiments.