# Uber's Smart Database Overload Fix _Uber evolved its database overload protection from static rate-limiting to an intelligent, priority-aware system called Cinnamon, enhancing stability and user experience._ **Updated:** 2026-08-22 **Published:** 2026-05-28 **Source:** https://www.startuphub.ai/ai-news/technology/2026/uber-s-smart-database-overload-fix --- Uber's massive infrastructure, supporting over 170 million monthly users, relies on in-house databases like Docstore and Schemaless. These systems, built on MySQL, handle petabytes of data and tens of millions of requests per second. At this scale, minor overloads can cascade, leading to system-wide failures. Ensuring fairness in a multitenant environment, where one user shouldn't hog resources, adds another layer of complexity. This [engineering challenge](https://www.uber.com/us/en/blog/from-static-rate-limiting-to-intelligent-load-management/) led Uber to develop an intelligent load management system. Massive Uber ScaleDriverFrom the articleUber's massive infrastructure, supporting over 170 million monthly users, relies on in-house databases like Docstore and Schemaless.Database Overload RiskDriverminor overloads can cascade, leading to system-wide failuresled toStatic Rate LimitingContextquota-based limits proved problematic and impreciseFrom the article 2 mentionsInitially, Uber experimented with quota-based rate limiting within its query engine.revealed limitationsConcurrency Load SignalContextreal load signal is concurrency, not just request countFrom the article 2 mentionsFurthermore, Cinnamon's Auto Tuner dynamically adjusts queue timeout thresholds and inflight limits using P90 latency and error rate signals, eliminating manual tuning and providing more nuanced load absorption than CoDel's static approach.informedEarly Load ManagementCoreCoDel and Scorecard techniques were initial stepsFrom the articleThis engineering challenge led Uber to develop an intelligent load management system.evolved toCinnamon SystemCoreintelligent, priority-aware load management systemFrom the article 9+ mentionsCinnamon considers request rank (priority) and dynamic system state to make smarter shedding decisions.resulted inEnhanced StabilityEffectimproved system stability and reliabilityFairness & User ExpEffectensuring fairness and better user experienceFrom the articleEnsuring fairness in a multitenant environment, where one user shouldn't hog resources, adds another layer of complexity. ## From Static Limits to Dynamic Control Initially, Uber experimented [with](/ai-news/ai-video/2025/ai-breaks-data-barriers-with-text-to-sql) quota-based rate limiting within its query engine. This involved assigning capacity costs to requests and enforcing fixed quotas. However, this approach proved problematic. It added complexity with external Redis calls and failed to accurately reflect the actual load on storage partitions. The cost model was imprecise, treating heavy and light queries similarly, and static quotas required constant stakeholder adjustments. A key insight emerged: overload management needed to be closer to the storage nodes themselves. The focus shifted to identifying more reliable signals than simple query-per-second (QPS) metrics. ## Concurrency: The Real Load Signal Concurrency, the number of operations in flight, proved a more dependable indicator of system load. It directly correlates with resource usage, following Little's Law (Concurrency = Throughput × Latency). This metric became central to detecting overload. ## Balancing Resilience and Fairness A core challenge in multitenant systems is balancing overall system resilience with per-tenant fairness. The system needed to shed low-priority traffic during global overload while also preventing individual 'noisy neighbors' from monopolizing resources. ## Early Load Management: CoDel and Scorecard The first iteration involved CoDel (Controlled Delay) queues, adapted from networking to manage request latency rather than just queue length. Separate queues for reads, writes, and slow operations were implemented. CoDel also introduced adaptive LIFO (Last-In, First-Out) behavior under pressure, favoring newer, more relevant requests over stale ones to fail fast and reduce wasted work. This approach improves responsiveness by shedding stale work and prioritizing fresh requests. To enforce per-tenant limits, the Scorecard engine was introduced. This rule-based system acts as a lightweight quota mechanism, preventing any single tenant from dominating shared infrastructure even under normal conditions. It aids in incident containment by isolating and capping misbehaving tenants without affecting others. Additional plug-in regulators, such as write byte limits and partition key throttling, were added to address subtler forms of overload not captured by concurrency alone. These regulators guard against issues like large write payloads or traffic skewed to specific partition keys. ## Limitations and the Evolution to Cinnamon Despite improvements, the CoDel system had limitations. It treated all requests equally, indiscriminately dropping low-priority and user-facing traffic, impacting customer experience. Fixed queue timeouts and static concurrency limits required frequent manual tuning, leading to operational toil and a 'thundering herd' problem on retries. This highlighted the need for dynamic and priority-aware load shedding. The solution came with Cinnamon, a priority-aware load shedder developed by Uber's Delivery team. Cinnamon considers request rank (priority) and dynamic system state to make smarter shedding decisions. Critical user-facing traffic (Tier 1) is protected over lower-priority asynchronous jobs (Tier 5). Cinnamon simplified the queue structure by marking long-running operations with lower priority, rather than using separate queues. This priority-aware shedding ensures that during overloads, critical user-facing requests are protected with minimal impact. Furthermore, Cinnamon's Auto Tuner dynamically adjusts queue timeout thresholds and inflight limits using P90 latency and error rate signals, eliminating manual tuning and providing more nuanced load absorption than CoDel's static approach. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory. © StartupHub.ai. All rights reserved. You may not republish this article in full without a license. Search engines and AI research tools may crawl and summarize for reference. Bulk reproduction or model training on this content requires a license. See https://www.startuphub.ai/terms.