From Netflix Tech Blog · 30 stories
New AI Models for Long-Horizon Coding Tasks Introduced
Several AI models aimed at long-horizon tasks in coding and robotics have been released. GLM-5.2 by Hugging Face extends support for coding-agent scenarios with a 1 million token context. Cognition's SWE-1.7 enhances long-horizon asynchronous tasks with reinforcement learning. Xiaomi-Robotics-1 combines vast pre-training data for improved robotics capabilities. These releases highlight advances in scaling and reasoning capabilities.
Kubernetes Extends Reach to Desktop Infrastructure, Highlighting Database Management Challenges
Kubernetes is being considered for managing desktop infrastructure, traditionally separate from cloud-native models. This shift aims to unify operational practices and lower costs related to outdated virtual desktop systems. However, while Kubernetes simplifies deployment, it also exposes complexities in managing databases, requiring expertise beyond standard DevOps skills.
Meta Introduces Hybrid Asset Classification for Privacy-Aware Infrastructure
Meta has unveiled a hybrid asset classification strategy using large language models (LLMs) to handle ambiguous data in privacy-aware infrastructure while maintaining deterministic rules for enforcement. This method addresses the complexities of AI-native products with varied data inputs, ensuring compliance and effective data governance. It is a response to the challenges posed by the increasing speed and scale of AI innovations, and the approach aims to better manage privacy controls for evolving AI products.
Netflix Observability Team Discusses Ontology-Driven Knowledge Graph for E2E Monitoring
Netflix's observability team presented on building an ontology-driven end-to-end knowledge graph to manage observability at their scale. This approach addresses the data engineering challenges of monitoring 65 million concurrent streams, 2 billion daily requests, and 38 million logging events per second. The knowledge graph aims to provide insights into user experience from application click to content playback.
Netflix Introduces GenPage AI to Optimize Personalized Homepages
Netflix has introduced GenPage, a generative AI model to create personalized homepages. GenPage improves user engagement and reduces latency by replacing the traditional multi-stage recommendation pipeline with a single model that uses user history and context. This shift could streamline content delivery and enhance personalized user experiences.
Netflix Introduces Pensive AI for Automated Workload Failure Diagnosis
Netflix developed Pensive AI to improve the diagnosis and remediation of failed data and machine learning workloads. This new system uses AI to extract errors and provide guidance, expanding coverage beyond the rule-based Pensive system.
Netflix Develops Automated Multi-Ratio Video Cropping System for Vertical Content
Netflix has developed and integrated an automated multi-ratio video cropping system into its internal production tools. This system generates high-quality vertical (9:16) and other aspect-ratio video variants from horizontal footage to support its mobile app feed and marketing campaigns, reducing manual reframing efforts.
Netflix Develops Quasar to Measure Impact of Library Changes Across Repositories
Netflix's Java Platform Team developed Quasar, a tool that indexes symbol definitions and references across their repositories. Quasar helps library producers understand the "blast radius" of API changes by identifying which repositories use specific symbols that are being altered or removed, enabling informed decisions before releasing potentially breaking changes.
Netflix implements tiered storage for TimeSeries data, moving older data to S3
Netflix has implemented two solutions for tiered storage of its TimeSeries data, moving older, less frequently accessed data from hot storage to Amazon S3. This change reduces storage costs while maintaining data availability for audits and investigations.
Netflix Rebuilds Ads Event Pipeline with Apache Flink for Live Streaming
Netflix re-engineered its Ads Event Publisher to support live streaming, de-duplication, and richer data signals. The update replaced a database lookup with real-time streaming joins in Apache Flink, enabling scalability for live content and removing a critical path dependency.
Netflix Conductor Workflow Engine Rebuilt for Scale After Open-Source Discontinuation
Netflix has rebuilt its Conductor workflow orchestration engine from the ground up to handle increased scale, following its decision to discontinue the open-source version. The internal fork now processes over a billion workflow executions quarterly, necessitating a re-architecture of its data plane, evaluation engine, and work allocation mechanisms.
Netflix Automates Container Right-Sizing for Serverless Media Processing Platform
Netflix developed the Stratum Resource Tuner (SRT) to automatically right-size containers on its internal serverless media processing platform, Stratum. This system addresses the challenge of manual resource over-provisioning and configuration drift, which leads to increased operational costs and cognitive load for engineers.
Netflix details GPU infrastructure reliability for ML training and inference
Netflix developed methods to detect failing GPUs and network fabric, automatically remediate machines, and allow long-running jobs to survive hardware loss. This system ensures continuous operation for machine learning models that power personalization and generative AI initiatives.
Netflix Details Workload Attestation for Apache Spark on Amazon EMR
Netflix developed a method to bridge cloud provider identities with internal identity systems for Apache Spark workloads on Amazon EMR. This allows managed compute instances, which initially only have cloud identities, to acquire internal identities necessary for accessing internal services and data. The solution enables consistent access control and auditing for data projects.
New Java CLI tools 'ja' and 'jig' introduced for module system development
A preview of 'ja' and its associated command-line tools, including 'jig' and 'jfmt', has been released to modernize Java development using the Java Module System. These tools aim to simplify module creation and consumption, particularly for coding agents, by providing composable command-line utilities for tasks like dependency resolution and code formatting.
Netflix implements multimodal embeddings for improved asset personalization at scale
Netflix has integrated multimodal embeddings into its personalization models to address the cold-start problem for new titles and assets. This allows models to understand visual and audio cues of assets, enabling personalized recommendations sooner after a title's launch by leveraging taste signals from related content.
Netflix Introduces GenRec, an LLM-Backed Recommendation System
Netflix developed GenRec, a new recommendation ranker that uses a post-trained internal foundation Large Language Model (LLM) to improve content recommendations. This system achieved statistically significant improvements in A/B tests compared to existing production models, while reducing reliance on extensive feature engineering.
Netflix Shares Insights on Building Real-Time Service Dependency Maps
Netflix engineers detailed challenges and architecture behind creating a real-time service dependency map. This approach improves troubleshooting and performance insights in complex distributed systems.
Netflix Research Focuses on AI for Enhanced Video Editing Control
Netflix is exploring generative AI techniques to improve video editing workflows for greater creative control. The initiative aims to mitigate issues such as unintended edits and unnatural scene physics, addressing specific challenges faced by video editors.
Netflix Incorporates Kueue for Batch Computing in Kubernetes Transition
Netflix has migrated from its custom Compute Managed Batch (CMB) solution to Kueue, a cloud-native job queueing system for batch workloads. This transition simplifies batch processing and enhances the efficiency of their Kubernetes-native compute infrastructure.
Netflix Develops Automated System to Validate Catalog Metadata
Netflix implemented an automated data canary system to validate catalog metadata, detecting corruption within 10 minutes. This system addresses gaps in resiliency highlighted by a previous incident where corrupted data affected streaming services.
Netflix Implements New Data Management Framework to Address ACL Issues
Netflix has introduced a new data management framework to improve access control and workload execution across its extensive data platform. This change aims to streamline permissions management and reduce operational disruptions related to organizational changes.
Netflix Optimizes Content Launch Planning with Data-Driven Insights
Netflix is utilizing data-driven insights to mitigate risks in content launch planning. By analyzing production schedules and gaps in asset delivery timelines, the company aims to enhance the accuracy of its launch preparation processes.
Netflix Enhances Data Movement Architecture for Cassandra to Iceberg
Netflix has centralized its Data Movement process with a new architecture that replaces the legacy Casspactor system. This upgrade improves efficiency for transferring data from Apache Cassandra to Apache Iceberg, critical for analytics and operational tasks.
Netflix Develops Framework for Personalized Notification System
Netflix has introduced a hierarchical framework for its notification system, balancing short-term engagement with long-term member experience. The system employs a 'slow' policy for strategic planning and a 'fast' policy for real-time message decisions, addressing engagement challenges inherent in personalized messaging.
Netflix develops model to measure personalized recommendation impact on viewing behavior
Netflix researchers developed a new model to measure the causal impact of personalized recommendations on member viewing behavior. This model separates the effect of recommendations from a member's underlying preferences, addressing the challenge that recommendations are targeted to existing demand.
Netflix Transitions from In-House to Open-Source Flink Autoscaler
Netflix is migrating its Apache Flink stream processing jobs from a proprietary autoscaling system to an open-source solution from the Apache Flink community. This transition addresses limitations of the in-house system, which was not designed for the complexity of newer workloads, and aims to consolidate operations on a single, more capable platform.
Netflix details gRPC execution API for querying its Real-Time Distributed Graph
Netflix published the third part of a series explaining how it built its Real-Time Distributed Graph (RDG), focusing on the design of a fast and flexible serving layer for efficient graph querying. This part describes how the company addresses the challenge of querying a constantly evolving, billion-edge graph to achieve sub-100ms responses across various workloads.
Netflix Details Device Capability Modeling for Feature Management and Analytics
Netflix developed a comprehensive device capability data model and integrated feature flags to manage features across its diverse device ecosystem. This system allows Netflix to understand hardware limitations, optimize user experience, and make data-driven decisions about feature deployment, improving feature penetration and innovation speed.
Netflix Engineer Details Spark Experimentation Strategy for Production Pipelines
A Netflix engineer developed a method to run and monitor multiple Apache Spark experiments concurrently to address performance bottlenecks in a critical data pipeline. This approach allowed for testing at full production scale, overcoming limitations of small dataset sampling.