LinkedIn has integrated Apache Celeborn as a shared remote shuffle platform across most of its Apache Spark and Apache Flink workloads. This deployment separates the storage of intermediate data from the compute infrastructure, a significant architectural shift from previous methods. The company processes over 300,000 Spark jobs daily, consuming approximately 200 petabyte-hours of compute.
The adoption of Apache Celeborn resulted in a substantial reduction in daily shuffle fetch failures, decreasing from approximately 750 to 20. Additionally, LinkedIn observed a 10% reduction in compute costs and a 14% improvement in application runtimes. These metrics indicate a more efficient and reliable data processing environment.
Previously, LinkedIn used Magnet, a push-based shuffle implementation, which aggregated shuffle blocks into larger files to reduce random disk reads. However, shuffle storage remained within the compute cluster, leading to resource contention between External Shuffle Services and compute workloads. Shuffle operations, which move intermediate data between processing stages, can stress networks, disks, and application runtimes.
With Apache Celeborn, map tasks push intermediate data to dedicated workers that aggregate and store partitions independently of compute executors. Reducers then fetch data directly from these workers. This architecture allows shuffle storage to scale independently of compute resources and reduces reliance on executor lifecycles. Celeborn operates as a separate shuffle service with dedicated workers and a replicated control plane, unlike Magnet's merger External Shuffle Services within the Spark compute cluster.
Operating Celeborn reliably across thousands of nodes and hundreds of thousands of daily applications required significant engineering effort beyond the upstream project. LinkedIn's deployment includes a five-node Raft-based control plane distributed across availability zones, utilizing Apache Ratis for metadata replication, leader election, and failover. The company also implemented network-aware replication and integrated Celeborn with its existing infrastructure.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
LinkedIn deployed Apache Celeborn as a shared remote shuffle platform for its Apache Spark and Flink workloads, separating intermediate data storage from compute infrastructure. This implementation reduced daily shuffle fetch failures from 750 to 20, lowered compute costs by 10%, and improved application runtimes by 14%. The change addresses inefficiencies in shuffle data handling, which previously caused resource contention within compute clusters.