RapidsMPF Achieves 1.8 TiB/s Data Shuffling for Out-of-Core Operations

RapidsMPF Performance

RapidsMPF, an out-of-core shuffler, has achieved a data shuffling speed of 1.8 TiB/s. This performance is significant for operations that require moving large datasets that do not fit into memory.

Addressing Shuffling Challenges

Shuffling is a core component in structured data analytics, essential for operations such as join, groupby, merge, and sort. These operations are often memory-intensive, can be slow due to data transport, and require synchronization, leading to potential Out-Of-Memory (OOM) errors or stalled execution plans.

RapidsMPF is designed to mitigate these issues by providing a reusable solution that manages data spilling to disk when memory limits are reached, allowing for budgeting of these operations.

Importance of Shuffling in Joins

Data joins, particularly in distributed systems, rely heavily on efficient shuffling. In-memory joins involve a build phase, where a hash table is constructed from one table, and a probe phase, where another table is scanned against this hash table. This process requires holding multiple tables and the hash table in memory simultaneously.

When data is distributed, the challenge intensifies as data may not be colocated, necessitating movement between processes or nodes. RapidsMPF's capabilities are particularly relevant for these distributed join scenarios.