Netflix Rebuilds Ads Event Pipeline with Apache Flink for Live Streaming

Ads Event Pipeline Evolution

Netflix has evolved its Ads Event pipeline, detailed in a two-part series. The initial Ads Event Publisher centralized ad telemetry collection, enrichment, and standardized data contracts for various downstream consumers like frequency capping, metrics, and billing. This system processes ad events while adhering to Netflix's privacy and security standards for member data.

Rebuilding for Live Streaming

The second phase of this evolution focused on supporting ads on live streams, improving de-duplication, and integrating richer data signals. This was achieved by rebuilding the Ads Event Publisher around stateful stream processing. The core change involved replacing a database lookup with a real-time streaming join using Apache Flink.

Technical Implementation and Benefits

This architectural shift allowed the system to scale for live content and removed a dependency from the critical path. The transition to Apache Flink was completed without downtime, despite the high-traffic nature of the pipeline. The enrichment process, which previously relied on the Ads Metadata Registry (a Key-Value store), now leverages Flink for real-time data correlation.

Enrichment Process Refinement

Previously, the ad server wrote impression metadata into the Ads Metadata Registry, and client devices received an opaque token. When the client fired an event with this token, the Ads Event Publisher would decode it and look up the metadata in the registry. The new Flink-based system integrates this enrichment step directly into the real-time streaming process, providing more immediate and scalable data correlation.