Hosting Incident #20003
We've received a report of an issue with Firebase Hosting.
Incident began at 2020-04-21 11:43 and ended at 2020-04-21 13:31 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| May 19, 2020 | 10:32 | Incident Analysis Impact: Most noncached requests made against the Hosting origin were returning a 503 instead of serving content between 2020-04-21 11:43 and 2020-04-21 13:31 (all times are US/Pacific). Root Cause: A large amount of a specific type of request traffic was not cached by our CDN for a single customer. All of these cache misses resulted in a huge spike in requests served by our origin service. Once this service was fully overloaded it began serving 503s to all customers. Resolution: The offending domain was blocked and appropriate preventative action was put in place to avoid similar issues in the future, examples including: additional log reporting, additional monitoring, and bug fixes in caching layer. |
|
| Apr 21, 2020 | 13:31 | The issue with Firebase Hosting has been resolved for all affected users as of Tuesday, 2020-04-21 12:39 US/Pacific. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we've worked on resolving the issue. |
|
| Apr 21, 2020 | 12:37 | Description: Mitigation work is currently underway by our engineering team. We have made a change and error rates are coming down quickly. We will provide more information by Tuesday, 2020-04-21 13:33 US/Pacific. Diagnosis: High 503 error rate for Firebase Hosting traffic. Workaround: None at this time. |
|
| Apr 21, 2020 | 11:58 | Description: We are investigating elevated 503 rates for Firebase Hosting traffic. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2020-04-21 13:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: High 503 error rate for Firebase Hosting traffic. Workaround: None at this time. |
|
| Apr 21, 2020 | 11:53 | Description: We are investigating elevated 503 rates for Firebase Hosting traffic. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2020-04-21 12:54 US/Pacific with current details. We apologize to all who are affected by the disruption. |
|
| Apr 21, 2020 | 11:47 | Description: We've received a report of an issue with Firebase Hosting as of Tuesday, 2020-04-21 11:43 US/Pacific. |