Improving Video Model Training Through Data Filtering and Annotation

The Shift to Data Quality in Generative Models

Recent advancements in image and video models are primarily attributed to improvements in data handling, rather than fundamental changes in model architecture. The prevailing wisdom has shifted from aggregating vast amounts of data to prioritizing the quality and relevance of training data.

Throwing low-quality data into pre-training can lead to models wasting capacity on mimicking undesirable data characteristics. Effective data filtering ensures models learn what is intended more efficiently.

Key Data Improvement Strategies

Three main approaches contribute to better model training: data filtering and rebalancing, data annotation, and synthetic data generation. Data filtering involves removing noisy data and strategically resampling datasets. Data annotation focuses on gathering richer captions, bounding boxes, and font details to help models distinguish visual concepts.

Synthetic data generation involves fine-tuning existing generative models to create training data for scenarios where natural data is scarce, such as image editing or reference-conditioning for specific model types.

Practical Application: Filtering on a Budget

One practical approach to data filtering, particularly for large datasets, involves using traditional computer vision algorithms on CPU clusters. This method avoids the need for extensive GPU resources, making the filtering process more cost-effective.

For video data, a crucial step is scene detection. Raw videos are segmented into clips at shot boundaries, as camera cuts are authorial decisions that a generative video model should not arbitrarily replicate. This pre-processing ensures that the model learns from meaningful video segments.