Uber's Zone Failure Resilience for OpenSearch

Uber has detailed its strategy for achieving Zone Failure Resilience (ZFR) in its OpenSearch deployments, a critical requirement for its distributed systems resilience. The goal is to ensure that core functions like querying and data ingestion remain operational even if an entire availability zone goes offline.

This resilience is built by integrating OpenSearch's native shard allocation awareness features with Uber's proprietary isolation group infrastructure. This system is underpinned by Odin, Uber's container orchestration platform, which is also a key component in achieving similar stability goals for other services.

Understanding Isolation Groups

Isolation groups act as a logical partitioning mechanism. They ensure that nodes within an OpenSearch cluster are distributed evenly across different failure domains, such as zones or racks. This prevents a single zone failure from disproportionately impacting the cluster.

A core principle is failure domain awareness, where isolation groups map directly to physical failure domains. This ensures that multiple isolation groups do not share the same zone without incurring scheduling penalties. Role-level balancing further guarantees that critical node roles are not concentrated in a single domain.

Stable group membership is also crucial. A node and its replacement are guaranteed to belong to the same isolation group, maintaining consistent shard placement semantics. Uber typically utilizes three isolation groups, meaning a single zone failure affects at most approximately 33% of the cluster's capacity.

Shard Allocation Awareness

OpenSearch's shard allocation awareness allows for the even distribution of shard copies across logical groupings of nodes. This is configured by defining custom node attributes and enabling awareness at the cluster level. For instance, with 5 total shard copies (1 primary, 4 replicas), OpenSearch distributes them across 3 isolation groups as 2, 2, and 1, ensuring the difference between any two groups never exceeds one copy.

However, this relies on a balanced pool of nodes. Uneven distribution can lead to unassigned shards. Therefore, even node distribution across isolation groups is a prerequisite for this system to function effectively.

Forced Shard Allocation Awareness for Stability

During a zone failure, OpenSearch's default behavior of aggressive shard rebalancing across remaining zones can destabilize the cluster. To counter this, Uber employs forced shard allocation awareness. The cluster is configured with the full set of expected attribute values.

When a zone fails and its associated isolation group disappears, OpenSearch recognizes the missing attribute values and refrains from over-allocating shards onto the remaining groups. Shards from the failed group remain unassigned, typically transitioning the cluster to a yellow state instead of triggering a cascade of rebalancing operations. This protects the surviving zones from excessive load, maintaining stability and performance.

This approach trades immediate full replication for cluster stability during failure events. Unassigned shards are only reallocated when the failed zone recovers or an administrator manually updates the configuration.

Resilience Against Zone and Node Failures

The most critical scenario involves a zone failure followed by an additional node failure. For data nodes, Uber ensures at least 3 copies of each shard are distributed across 3 isolation groups. A zone failure removes one group and one copy, and a subsequent node failure may remove a second copy, leaving the third available in the remaining group.

Cluster manager node resilience is handled by running 5 nodes instead of the typical 3. If a zone failure removes up to two cluster manager nodes, the cluster automatically shrinks its voting configuration to the remaining three nodes. A new primary leader is elected, and even if another node fails, the remaining two provide sufficient quorum for continued operation. This is vital for maintaining the availability required for disaster recovery for databases.

Uber's implementation of isolation groups has resolved previous issues where uneven node counts in physical zones led to yellow cluster states. By ensuring an equal number of nodes per isolation group, Uber guarantees 100% shard assignment and a green cluster health, eliminating disk skew and hot nodes caused by physical zone asymmetry.

This architecture, combining isolation groups for physical placement and shard allocation awareness for logical placement, decouples OpenSearch's resilience from the complexities of physical data center topology. It ensures Uber's search and analytics infrastructure remains stable, performant, and available, even under severe failure conditions like a zone plus one node failure.