Go Stack Allocation Boosts Uber CPU Efficiency

Uber is squeezing more performance out of its massive Go deployment. The ride-sharing giant has detailed how tweaking Go's stack allocation mechanisms slashed CPU usage by as much as 10% in key services, translating to significant operational cost reductions across millions of cores. This breakthrough, reported by Uber Engineering, underscores the critical role of granular optimizations for large-scale cloud infrastructure.

Go’s runtime, designed for efficiency, uses goroutines which have much smaller initial stack sizes (2KB) compared to OS threads (2MB). When a goroutine's stack fills up, Go dynamically doubles its size and copies the data—a process that consumes CPU cycles. At Uber's scale, where 1% efficiency gains equate to millions of dollars, minimizing this 'stack expansion' is paramount.

The Stack Expansion Problem

While Go 1.19 introduced adaptive stack sizing to improve initial allocation, it wasn't enough for all workloads. Uber found that repeated stack growth remained a significant CPU drain in some services.

One service, in particular, showed nearly 10% of its CPU consumption attributed to stack growth, despite having ample memory available.

Tuning the Runtime

Uber explored two main paths: goroutine pooling and customizing the Go runtime. Goroutine pooling requires substantial code changes and introduces its own overhead. Instead, Uber opted to modify the Go runtime directly.

By disabling Go's adaptive stack sizing and stack shrinking features, and then pre-allocating goroutine stacks to a static, larger size, they aimed to eliminate the costly dynamic expansion process. This involved patching the Go source code to expose and control internal variables related to stack size.

The team developed an analysis tool that inspects Go binaries to determine the actual stack usage of functions. This allowed them to statically set optimal stack sizes, preventing runtime overrides and minimizing the performance impact.

Impact and Future

The results were dramatic. One service saw its stack growth CPU cost drop from nearly 10% to under 1% after increasing the stack size from the default 2KB to 32KB. Memory usage increased, but remained well within container limits.

Uber plans to expand this optimization across more services, focusing on those with high CPU usage from stack growth and relatively low memory footprints. This approach, while requiring internal runtime modifications, demonstrates the potential for significant gains by optimizing core language features for specific operational demands.

This effort highlights how even mature platforms like Go can benefit from deep, specialized tuning for extreme scale. This focus on CPU efficiency in Go services is reminiscent of other efforts, such as mimalloc: Microsoft's Speed Boost for Apps.