Replacing mmap with io_uring in a Rust query engine resulted in slower performance

Initial Performance Issues with mmap

Conviva's query engine, built with Rust, DataFusion, and Arrow, initially used `mmap` for reading large Arrow IPC files. This approach was chosen for its convenience and zero-copy random access capabilities, which aligned well with the Arrow IPC format. However, under heavy concurrent query loads in production, `mmap` exhibited significant performance problems. The system experienced OS page cache shrinkage, numerous page faults, and p95 latencies spiking from approximately 30 seconds to over 150 seconds. Adding more pods exacerbated the issue, suggesting `mmap` page-cache thrashing under memory pressure.

Testing the mmap Hypothesis

To isolate the problem, Conviva conducted a controlled benchmark comparing one pod against four pods on the same host with identical concurrent query loads. The expectation was that four pods would offer better parallelism and isolation. However, the results showed the opposite: a single pod using `mmap` performed significantly better, being up to 41% faster at maximum and over 20% faster at p95 for 14-day queries. This outcome challenged the initial assumption that `mmap` was the direct cause of the performance bottleneck, as the shared `mmap` page cache across pods did not lead to CPU contention.

Unexpected Results with io_uring

Despite the benchmark results, the team proceeded with replacing `mmap` with `io_uring`, anticipating an improvement in performance. However, this change unexpectedly led to a slower query engine. This indicates that the initial diagnosis of `mmap` being the primary cause of performance degradation under heavy load was incorrect, and the underlying issues were not resolved by switching to `io_uring`.