Uber Develops Context-Aware Mechanism to Mitigate Retry Storms in Distributed Systems

Addressing Retry Storms in Distributed Systems

Uber has implemented a new context-aware mechanism in its shared infrastructure to combat retry storms. These storms historically affect business operations and brand trust by escalating localized service outages into wider system failures.

Traditional retry configurations and budgets, while providing some mitigation at the service level, often lack visibility into how errors propagate and amplify across complex dependency chains and fan-out patterns. This makes it difficult to protect the infrastructure from a domino effect triggered by a single service failure.

Limitations of Current Retry Approaches

A key challenge with existing retry behaviors is their lack of context awareness. Current systems can control the number of retries but not precisely when they occur, struggling to differentiate between errors generated by a service and those merely propagated through it. Consequently, retries are applied uniformly rather than conditionally.

This uniform approach is effective for transient or low-rate failures. However, during moderate or severe degradation, it becomes counterproductive. Aggressively retrying against an already struggling service increases its load, accelerates its failure, and amplifies retry traffic across upstream dependencies, leading to broader incidents.

Uber's Context-Aware Solution

While translating downstream error codes upstream for retry context is theoretically possible, Uber found this approach unscalable due to its large fan-in and fan-out, evolving call flows, and the need for frequent adaptive changes. To overcome these limitations, Uber developed a context-aware mechanism.

This new mechanism, integrated into shared infrastructure, handles errors more efficiently by providing precise control over when retries occur, preventing the uniform application of retries that can exacerbate system stress during outages.