Zalando Implements In-Process Client-Side Load Balancer for High-Throughput API

Client-Side Load Balancing for High Fan-Out Traffic

Zalando's engineering team implemented an in-process, client-side load balancer to manage high-throughput traffic for its Product Read API. This API processes around 1 million requests per second and serves 25 markets for the online fashion retailer. The primary motivation was to address unpredictable latency and improve operational visibility for batch endpoints that fan out to up to 100 parallel calls.

Addressing Latency and Cost Challenges

Previously, batch requests relied on Skipper, a shared cluster edge load balancer, which introduced latency spikes that were difficult to distinguish from internal issues. By moving the routing decision for high fan-out internal traffic into the calling process, Zalando gained more control. This shift resulted in more predictable latency, a reduction in infrastructure costs, and clearer insights into the actual sources of failures.

Reimplementing Consistent Hashing

The new client-side load balancer reimplements Skipper's exact consistent hashing algorithm, using xxHash64 with 100 virtual nodes per endpoint. This ensures that during migration, both the old and new paths route identical keys to identical pods, preventing cache splits. This design minimizes cache churn when endpoints are added or removed, as only about 1/N of keys are remapped.

Rollout Strategy and Operational Improvements

The implementation involved several engineering decisions, including occupancy-based load balancing, cache-aware scaling, and experiments with availability-zone routing. Zalando replaced control-plane-crashing polling with a watch-based Kubernetes informer and rebuilt a slow pipeline into fast, reversible deployments. The load balancer was rolled out gradually from 1% to 100% using feature toggles, which allowed Zalando to shrink its Skipper fleet from over 50 pods to 8 and reduce daily deployment costs.