Netflix Develops Automated Multi-Ratio Video Cropping System for Vertical Content

Automating Vertical Video Production

Netflix has implemented an automated multi-ratio video cropping system to address the growing need for vertical video content. This system is designed to convert traditional horizontal cinematic footage into mobile-friendly aspect ratios, such as 9:16, 4:5, and 1:1, with minimal manual intervention. The development responds to the increasing importance of vertical formats for Netflix's social media presence, which has 1.2 billion followers, and the recent rollout of its in-app vertical video feed, Clips.

Intelligent Cropping Approach

The system treats video resizing as a tracking and temporal optimization problem, aiming to create crops that maintain the original shot's intentionality. It operates in three main stages: shot detection and splitting, and focus region discovery. This pipeline processes assets in a single pass to generate multiple aspect ratios without additional computational overhead.

Shot Detection and Focus Region Identification

The first stage involves segmenting video into atomic shots using a neural-network-based shot boundary detector. For shots with multiple conversing characters, an Active Speaker Detection approach logically splits continuous shots into sub-shots, allowing the virtual camera to refocus on different speakers. This active speaker detection uses an internal face detector and LipFlap. Following shot definition, the system identifies the most important region in each frame by combining face detections with visual saliency maps.

Impact on Content Production

By automating the video resizing process, Netflix can scale its production of vertical assets for both its internal mobile app feed and external marketing campaigns. This reduces the previous reliance on manual reframing by editors and producers, which was not scalable given the expanding content catalog and use cases.