Ansible Automation Powers Uber's Global Network

Uber's sprawling corporate network, connecting thousands of engineers across six continents, demands robust automation. To maintain seamless uptime and handle daily operational updates, the ride-hailing giant relies heavily on Ansible automation. This approach is critical for orchestrating cloud and on-premises infrastructure, enabling continuous deployments, and managing multi-vendor hardware globally.

Infrastructure automation is no longer a luxury but a necessity for large distributed systems. Uber utilizes various Infrastructure as Code (IaC) tools, employing Terraform for cloud provisioning and Puppet for configuration management on running machines. However, for on-premises network hardware configuration, Ansible has become the go-to solution for enforcing stable network standards.

The Rise of Ansible

Born from a desire to simplify server management, Ansible emerged in 2012 as an open-source project. Its agentless design, requiring only SSH access to clients, and its reliance on Python and YAML made it significantly easier to adopt than contemporaries like Chef and Puppet. This simplicity quickly attracted network and DevOps engineers, expanding its use from Linux servers to orchestrating complex network gear.

Red Hat's stewardship and the development of Ansible Collections, which allow for vendor-specific modules (e.g., for Cisco, Juniper, Arista), further solidified its position. This modularity and community-driven enhancement made Ansible the preferred choice for Uber's network configuration needs.

Scaling Network Operations Globally

Managing a network of Uber's scale, encompassing around 5,000 devices across five global regions, presents significant challenges. Hardware diversity is a constant, with different offices and data centers requiring varied equipment. Ansible helps by standardizing configuration management across generic switches, gateways, and firewalls, regardless of vendor.

The core of Uber's strategy lies in its Ansible inventory, which groups hosts by region, functionality, and operating system. These groups, defined in YAML files, specify desired configuration states. Ansible decouples configuration logic from vendor-specific syntax, translating generic variables into the appropriate commands for each device.

Any manual changes not reflected in the Ansible inventory are overwritten nightly, a process dubbed "Daily Nightly Enforcement." This ensures network stability, provides a versioned record of the network state, and maintains a clean configuration.

Daily Nightly Enforcement Workflow

This critical workflow involves three main steps managed by Ansible playbooks.

First, a backup playbook captures the current configuration of all devices and stores it in a private GitHub repository, enabling change tracking.

Second, config generation uses Ansible playbooks to gather device facts and render vendor-specific configurations from Jinja templates. These "golden configurations" are then processed to mask sensitive data and prepared for storage.

Finally, a nightly push playbook deploys these golden configurations to the devices. Failures at any stage trigger alerts, feeding into Uber's broader observability strategy.

This automated process covers everything from BGP communities and DHCP server lists to VLAN management. It also facilitates tasks like automated password rotation and Zero Touch Provisioning (ZTP) for new devices, significantly reducing manual effort for Global Network Operations teams.

Navigating Automation Challenges

Despite its power, Ansible automation carries risks. Errors can have a wide blast radius, affecting thousands of devices. Managing multi-vendor environments requires careful structuring of Ansible code to avoid duplicated variables and vendor-specific syntax issues.

Handling large configuration files, often thousands of lines long, necessitates tools to clean up formatting and prevent syntax errors. Searching for issues in extensive repositories of backups and generated configurations also demands efficient management, including regular removal of legacy hosts.

When public Ansible collections or vendor modules fall short, Uber develops custom Ansible and Python modules. This requires dedicated engineering resources and ongoing maintenance. Furthermore, integrating with vendors not fully supported by Ansible necessitates orchestrating Ansible code with other management portals.

Ultimately, Ansible automation is the linchpin enabling Uber's global network operations. It provides the confidence to manage diverse network vendors and regions, transforming complex tasks into a streamlined, automated workflow.