Uber Enhances OpenSearch Resilience During Zone Outages

Overview of Uber's Approach to Zone Outages

Uber has detailed its strategies for maintaining OpenSearch deployments during zone outages. This methodology centers on utilizing OpenSearch's built-in shard allocation in conjunction with an isolation-group system, supported by the Odin container orchestration platform. Such a setup ensures that both query and ingestion capabilities remain functional even in the event of node failures.

Isolation Groups and Their Role

The implementation of isolation groups (IGs) serves as a logical layer that assists in managing physical failure domains relative to OpenSearch's placement logic. Each isolation group is designed to maintain an equal number of nodes, thus mitigating issues stemming from uneven zone capacities and preventing the occurrence of unassigned shards. For most of its services, Uber operates with three isolation groups, ensuring that the loss of one zone affects no more than one-third of the capacity.

Handling Failures Effectively

A significant challenge arises during a failure scenario, as OpenSearch typically rebalances quickly across remaining nodes when it detects missing shard copies. This rapid response can lead to excessive strain on resources. Uber addresses this by implementing forced shard allocation awareness, allowing the cluster to maintain its configuration without reallocating shards to immediate physical nodes. As a result, the cluster transitions to a yellow state without triggering a rebalancing overload. Shards are only reassigned once nodes are restored or configurations updated.

Cluster Management Enhancements

To bolster resilience, Uber utilizes five cluster manager nodes rather than the standard three. This adjustment employs OpenSearch's automatic voting configuration, which allows for effective election of a new primary node even when two out of five nodes fail. This mechanism assures continuous quorum and stable operation of the OpenSearch cluster during adverse conditions.