Uber Modernizes Artifact Storage

Uber's sophisticated engineering ecosystem, built on vast monorepos and thousands of microservices, hinges on reliable storage and distribution of build artifacts. For over a decade, a centralized on-prem platform managed this critical path, handling dependency resolution and artifact storage. This legacy system, deployed across two data centers, utilized local disks and a complex replication strategy.

However, exponential growth in scale and artifact volume exposed significant limitations. Disk space constraints led to manual, high-risk storage rebalancing. Inconsistent, asynchronous replication often resulted in silent failures, leaving artifacts vulnerable to data loss. Hardware failures demanded tedious manual data evacuation, a process fraught with operational risk.

Software upgrades were particularly perilous, involving multi-terabyte database migrations and extensive manual coordination, with the potential for complete cluster failure.

Shifting to a Managed Platform

To overcome these hurdles, Uber opted for a managed Software-as-a-Service (SaaS) platform. This move offered built-in regional isolation, high availability, and asynchronous cross-region replication, fundamentally shifting operational burdens like upgrades and security patches to the provider. Cloud-native blob storage replaced local disks, eliminating capacity concerns and mitigating hardware failure risks.

Tackling Cloud Egress Costs with a Proxy Layer

The transition to a cloud-based solution introduced a new challenge: significant data egress costs. With monthly artifact downloads exceeding 5 petabytes, Uber needed a way to reduce repeated data transfers without compromising the SaaS platform's role as the source of truth. The solution was an internal proxy layer.

This proxy acts as a validation layer, not a traditional cache. It forwards client requests to the SaaS origin but uses conditional headers like If-None-Match and If-Modified-Since. This prevents re-downloading unchanged artifacts, drastically cutting egress. The proxy maintains an artifact metadata store to track checksums and timestamps, ensuring correctness is always validated against the authoritative source.

Building a purpose-designed validation proxy was necessary because generic HTTP caching proxies, reliant on TTLs, lacked the strong consistency guarantees and per-request validation Uber required.

New Architecture Delivers Results

The new proxy-based architecture, deployed across two Uber data centers for low latency and active-active resilience, has yielded substantial improvements. Egress costs have been slashed by over 99% by avoiding redundant downloads. Overall reliability now stands at 99.99% at the proxy layer.

This approach ensures cost growth is tied to new or updated artifacts, not repeated downloads, reducing overall artifact-related egress costs by nearly 90%. Latency remains comparable to the legacy system, as unchanged artifacts are served directly from the internal cache after a lightweight validation check.

Enhanced observability provides full visibility into every request, making it easier to detect and address issues like replication gaps or failures quickly.

Ensuring Resilience and Future Optimization

The proxy is designed for fail-safe operation. In case of internal dependency failures, it can fall back to direct passthrough to the SaaS origin. Load balancing handles instance failures transparently, and GeoDNS manages failover for data center or cloud region outages.

Crucially, the proxy never serves stale artifacts; conditional requests ensure cached content is only used when explicitly confirmed as current by the SaaS platform. Failed downloads are discarded, preventing corrupted data.

Next steps involve enhancing the cache layer to support range requests, optimizing for large objects (>8GB), introducing node-local hot caches for bursty access, and implementing request coalescing to manage peak load thundering-herd behavior.

This Uber artifact storage modernization reflects a strategic shift towards managed services, augmented by intelligent internal infrastructure to balance cost, performance, and reliability for critical software build dependencies.