Netflix details GPU infrastructure reliability for ML training and inference

Maintaining GPU Health for ML Workloads

Netflix has implemented a system to ensure the reliability of its GPU infrastructure, which is critical for machine learning training and inference. This system addresses the challenge of hardware failures during long-running jobs that can span weeks across hundreds of GPUs and machines. Such jobs require all components to function continuously, as a single failure can halt progress.

Automated Detection and Remediation

The platform includes mechanisms to detect hardware failures, specifically in GPUs and network fabric, both before and during job execution. Upon detection, the system automatically remediates the affected machines without human intervention. This automation is crucial for maintaining the integrity of large-scale machine learning operations.

Job Survival During Hardware Loss

A key feature of Netflix's approach is enabling long-running machine learning jobs to continue even if a machine fails. This capability is vital for training models that underpin personalization features and generative AI advancements, which are increasingly central to Netflix's operations. These training workloads operate on a Kubernetes-native Compute platform.

Business Impact of GPU Reliability

GPU capacity is a rapidly growing and expensive part of Netflix's infrastructure. Ensuring its reliability directly impacts business availability, as disruptions to training runs or inference services can affect member experience and content production. The continuous operation of these systems supports Netflix's investment in generative AI for recommendations, promotional assets, and creative tools.