Pinterest Implements Event Time Watermarks for Petabyte-Scale Data Partition Finalization

New Data Finalization Mechanism

Pinterest has implemented an event time-based partition finalization mechanism within its next-generation database ingestion framework. This system enables downstream data processing jobs to ascertain when streaming data partitions are sufficiently complete for consumption. The core challenge addressed is the arrival of data records after a partition's wall clock interval has concluded in continuous Change Data Capture (CDC) ingestion.

Framework Components and Scale

The ingestion framework at Pinterest integrates CDC, Kafka, Flink, Spark, and Iceberg to transfer database changes from MySQL, TiDB, and KVStore into analytical tables. This platform manages petabyte-scale data and thousands of pipelines, with database changes typically becoming available for analysis within 15 minutes to an hour. Unlike previous batch-based methods where job completion signaled partition readiness, streaming ingestion continuously commits changes without a clear completion point.

How the Watermark System Works

The new mechanism operates within the Flink to Iceberg sink. For each CDC record, a configured event time is extracted. During Flink checkpoints, the system gathers minimum, maximum, and percentile event time statistics, which are then stored in the Iceberg snapshot summary. After an Iceberg commit, a partition finalization watermark is calculated from recent checkpoint statistics. The default policy uses the minimum event time across a configurable number of recent commits, rounded down to the nearest hour. This watermark is monotonic, preventing regressions from late-arriving records.

Metadata Storage and Downstream Access

Pinterest stores the finalization watermark as an Iceberg table property. This allows downstream sensors to read the watermark directly without requiring a separate coordination service. Spark also propagates this metadata to the base table during its CDC upsert commit. This design signifies that partition finalization is no longer a side effect of a job finishing, but an explicit, measurable state.