Uber Automates Service Mesh Dependency Analysis for Failure Propagation

Automated Dependency Mapping

Uber has implemented an automated system to analyze service dependencies and understand how failures propagate across its service mesh. This system utilizes RPC middleware to correlate failures between callers and callees during production requests. It classifies dependencies into three categories: fail-close (failure consistently causes caller failure), fail-open (failure does not cause caller failure), or unknown (insufficient evidence).

Addressing Microservices Complexity

In a microservices architecture with thousands of services, identifying the root cause of a request failure can be complex due to deeply nested and asynchronous service calls. Uber's new approach helps to quickly pinpoint which dependency contributed to a failure, providing faster feedback than traditional distributed tracing methods, especially for rare failure events.

Limitations of Distributed Tracing

Uber initially considered distributed tracing but found its sampling rate too slow for diagnosing infrequent failures. With over 99.9% of requests completing without errors, a 0.01% sampling rate meant capturing a rare erroneous request could take several hours. The new system prioritizes faster dependency analysis over individual request visibility for these scenarios.

Implementation Details

The analysis is integrated into YARPC, Uber's RPC framework for Go and Java, through inbound and outbound middleware. This allows for precise dependency correlation by generating request IDs and correlating them across service calls. While Uber's Muttley proxy handles service-to-service requests, it lacked the necessary caller endpoint context, leading to the RPC framework integration.