Cloudflare Reduces Memory Footprint by 100 TB with Consistent Hashing Optimization

Memory Optimization in Load Balancing

Cloudflare announced a significant reduction in the memory footprint of its internal load-balancing service. By optimizing the Rust implementation of consistent hashing within pingora-ketama, the company reclaimed more than 100 TB of RAM. This optimization targeted the data structures used for distributing cacheable requests across servers.

Addressing Excessive Memory Usage

Cloudflare's Pingora Backend Router relies on consistent hashing, but supporting weighted servers and various feature combinations led to a proliferation of hash rings, increasing memory consumption. The team identified that the pingora-ketama consistent-hashing data structures were consuming excessive memory, prompting an investigation into their design.

Statistical Analysis and Hash Reduction

Engineers applied probability theory to determine the minimum number of hash points required for accurate distribution. This analysis allowed them to reduce the number of hashes per server by approximately 90% without compromising load-balancing accuracy. The team found that adding more hashes yielded diminishing improvements in load distribution, with the final 90,000 hashes improving error by only 0.7%.

Data Structure Compaction

Further memory savings came from optimizing the Rust data structures that stored the remaining hashes. The original representation used an 8-byte 'Point' structure, but the hash and server index could be compactly represented in just 6 bytes. This compaction contributed to the overall memory reduction.