Google Cloud last week experienced a major widespread outage, and days later released a detailed report sharing why it failed.
Multiple Google Cloud and Google Workspace products experienced increased 503 errors in external API requests, impacting customers.
In a status report, the company blamed the occurrence on an “invalid automated quota update to our API management system which was distributed globally, causing external API requests to be rejected.”
To recover, Google bypassed the “quota check” allowing it to recover in most regions within two hours, but the quota policy database in us-central1 became overloaded, resulting in much longer recovery in that region.
Its customers had intermittent API and user-interface access, but Google said existing streaming and Infrastructure as a Service -- a cloud computing model that provides on-demand access to computing resources like servers, storage, and networking over the internet -- were not impacted.
advertisement
advertisement
Cloud computing has become important or more important than search and ad services because it has become the backbone of Google’s business. Since May, the company Google’s cloud division has signed
Since May 1, 2025, Google Cloud has signed several significant deals related to AI and expanding cloud infrastructure.
OpenAI signed a deal with Google Cloud for additional computing capacity for AI model training and operation. The partnership seeks to reduce OpenAI's reliance on one cloud vendor and increase the availability of computing resources.
Google Cloud has also announced a partnership with Databricks, integrating Gemini models into the Databricks Data Intelligence Platform.
Amadeus and Google Cloud partnered to advance cloud-based operations and AI innovation in the travel industry.
When Google Cloud failed, several products had moderate residual impact for up to one hour after the primary issue was mitigated, and a small number were recovering after that.
Google completed an Incident Report that described the root cause. “We deeply apologize for the impact this outage has had. Google Cloud customers and their users trust their businesses to Google, and we will do better,” the company wrote in a blog post. “We apologize for the impact this has had not only on our customers’ businesses and their users but also on the trust of our systems. We are committed to making improvements to help avoid outages like this moving forward.”
Despite all the technology, experience and knowledge, Google will need to do better because unplugging the internet is no longer an option.
Too many companies rely on it for support, and it has too many competitors to make these types of mistakes.
“This code change and binary release went through our region by region rollout, but the code path that failed was never exercised during this rollout due to needing a policy change that would trigger the code,” Google’s report explains. A red button had been installed to turn off that particular policy serving path, but it was never used.
The change "did not have appropriate error handling nor was it feature flag protected. Without the appropriate error handling, the null pointer caused the binary to crash. Feature flags are used to gradually enable the feature region by region per project, starting with internal projects, to enable us to catch issues. If this had been flag protected, the issue would have been caught in staging," according to the report.
Last Thursday, Google made a policy change, inserting it into what the company calls "the regional Spanner tables that Service Control uses for policies." The report provides specific details.
Within two minutes, Google's Site Reliability Engineering team triggered the incident. Within 10 minutes, the root cause was identified and the red-button (to disable the serving path) was being put in place. The red-button was ready to roll out about 25 minutes from the start of the incident. Within 40 minutes of the incident, the red-button rollout was completed, and we started seeing recovery across regions, starting with the smaller ones first.