Netflix Introduces Pensive AI for Automated Workload Failure Diagnosis

Addressing Workload Failures at Scale

Netflix's data and machine learning practitioners execute millions of workloads daily across various systems like Titus, Spark, Trino, and Druid. On peak days, approximately 100,000 workloads fail. Diagnosing these failures requires engineers to analyze logs, determine the cause, and decide on a course of action, consuming significant engineering time and resources.

Evolution from Pensive to Pensive AI

Netflix initially developed Pensive, an auto-diagnosis and remediation system, to address this issue. Pensive collected logs, used regular expressions and Java rules to classify errors, and provided guidance. This system made about 50% of failures self-service or automatic.

However, rule-based systems require manual updates for new failure modes and log format changes. Over one three-month period, roughly half of the seven million job-failure classifications processed by Pensive fell into an 'unknown' category, indicating a need for broader coverage.

Pensive AI's Approach and Impact

To expand coverage, Netflix invested in Pensive AI, integrating AI-assisted error extraction and focused diagnosis. The system wraps AI around deterministic processes to ensure broader coverage, low latency, controlled AI costs, and easier maintenance. This design allows for predictable evaluation inputs and responsive product behavior.

Early results show that AI error extraction now covers 100% of failures, including those not handled by the legacy Pensive system. This development aims to provide more comprehensive and reliable failure diagnosis for Netflix's data and ML practitioners.