← All stories
● Covered by 1 source · 1 reportMedium impact1 positive

LinkedIn Reduces Spark Shuffle Failures by 97% Using Apache Celeborn

🔄 Updated 3d ago
New to BrevFeed? We gather this story from every outlet covering it into one summary — ranked by real-world impact, not just the latest headline — so you never miss what matters. What is BrevFeed? →

Key points

  • Daily shuffle fetch failures dropped from 750 to 20.
  • Compute costs decreased by 10%.
  • Application runtimes improved by 14%.
  • Celeborn separates shuffle storage from compute executors.

Apache Celeborn Deployment

LinkedIn has integrated Apache Celeborn as a shared remote shuffle platform across most of its Apache Spark and Apache Flink workloads. This deployment separates the storage of intermediate data from the compute infrastructure, a significant architectural shift from previous methods. The company processes over 300,000 Spark jobs daily, consuming approximately 200 petabyte-hours of compute.

Operational Improvements

The adoption of Apache Celeborn resulted in a substantial reduction in daily shuffle fetch failures, decreasing from approximately 750 to 20. Additionally, LinkedIn observed a 10% reduction in compute costs and a 14% improvement in application runtimes. These metrics indicate a more efficient and reliable data processing environment.

Addressing Shuffle Inefficiencies

Previously, LinkedIn used Magnet, a push-based shuffle implementation, which aggregated shuffle blocks into larger files to reduce random disk reads. However, shuffle storage remained within the compute cluster, leading to resource contention between External Shuffle Services and compute workloads. Shuffle operations, which move intermediate data between processing stages, can stress networks, disks, and application runtimes.

Architectural Shift with Celeborn

With Apache Celeborn, map tasks push intermediate data to dedicated workers that aggregate and store partitions independently of compute executors. Reducers then fetch data directly from these workers. This architecture allows shuffle storage to scale independently of compute resources and reduces reliance on executor lifecycles. Celeborn operates as a separate shuffle service with dedicated workers and a replicated control plane, unlike Magnet's merger External Shuffle Services within the Spark compute cluster.

Engineering Challenges and Solutions

Operating Celeborn reliably across thousands of nodes and hundreds of thousands of daily applications required significant engineering effort beyond the upstream project. LinkedIn's deployment includes a five-node Raft-based control plane distributed across availability zones, utilizing Apache Ratis for metadata replication, leader election, and failover. The company also implemented network-aware replication and integrated Celeborn with its existing infrastructure.

✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →

The daily brief

One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.

One email a day. Unsubscribe in one click, any time.

Today's brief

Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.

~4 min · 3 stories · Nov 21

▶ Play today's brief Listen on Spotify

New every morning, and the back catalogue is archived by date.

Primary sources

GitHub apache/celeborn

Reporting from

LinkedIn deployed Apache Celeborn as a shared remote shuffle platform for its Apache Spark and Flink workloads, separating intermediate data storage from compute infrastructure. This implementation reduced daily shuffle fetch failures from 750 to 20, lowered compute costs by 10%, and improved application runtimes by 14%. The change addresses inefficiencies in shuffle data handling, which previously caused resource contention within compute clusters.