GitHub has acknowledged a series of significant availability and performance issues that have plagued its platform in recent weeks. The company, a cornerstone for developers worldwide, has failed to meet its own reliability standards, impacting workflows and user confidence. According to a blog post, the most disruptive incidents occurred on February 2, February 9, and March 5.
These outages stem from a confluence of factors, primarily rapid usage growth straining existing architecture. This growth exposed scaling limitations and architectural coupling, which allowed isolated problems to spread across critical services. A key contributing factor was the system's inability to adequately shed load from misbehaving clients.
February 9 Incident: A Cascade of Issues
The February 9 incident, in particular, highlighted these vulnerabilities. A core database cluster responsible for authentication and user management became overloaded. This was exacerbated by the release of two popular client-side applications with unintentional changes leading to a tenfold increase in read traffic.
A hastily adjusted cache refresh TTL (Time To Live) from 12 hours to 2 hours, implemented to expedite a new model release, compounded the problem. While weekend load masked the issue initially, the start of the work week, coupled with the client app updates and another model release, overwhelmed the database. The complexity of the issue, involving both increased write volume from the TTL change and read volume from the client apps, prolonged the incident.
Further complicating the response was a lack of granular controls to block excessive traffic further up the stack, a problem exacerbated by cascading service interactions.