Hosting Incident #20003

We've received a report of an issue with Firebase Hosting.

Incident began at 2020-04-21 11:43 and ended at 2020-04-21 13:31 (all times are US/Pacific).

Date Time Description
May 19, 2020 10:32

Incident Analysis

Impact: Most noncached requests made against the Hosting origin were returning a 503 instead of serving content between 2020-04-21 11:43 and 2020-04-21 13:31 (all times are US/Pacific).

Root Cause: A large amount of a specific type of request traffic was not cached by our CDN for a single customer. All of these cache misses resulted in a huge spike in requests served by our origin service. Once this service was fully overloaded it began serving 503s to all customers.

Resolution: The offending domain was blocked and appropriate preventative action was put in place to avoid similar issues in the future, examples including: additional log reporting, additional monitoring, and bug fixes in caching layer.

Apr 21, 2020 13:31

The issue with Firebase Hosting has been resolved for all affected users as of Tuesday, 2020-04-21 12:39 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we've worked on resolving the issue.

Apr 21, 2020 12:37

Description: Mitigation work is currently underway by our engineering team.

We have made a change and error rates are coming down quickly.

We will provide more information by Tuesday, 2020-04-21 13:33 US/Pacific.

Diagnosis: High 503 error rate for Firebase Hosting traffic.

Workaround: None at this time.

Apr 21, 2020 11:58

Description: We are investigating elevated 503 rates for Firebase Hosting traffic. Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2020-04-21 13:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: High 503 error rate for Firebase Hosting traffic.

Workaround: None at this time.

Apr 21, 2020 11:53

Description: We are investigating elevated 503 rates for Firebase Hosting traffic. Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2020-04-21 12:54 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Apr 21, 2020 11:47

Description: We've received a report of an issue with Firebase Hosting as of Tuesday, 2020-04-21 11:43 US/Pacific. We will provide more information by Tuesday, 2020-04-21 12:15 US/Pacific.

All times are US/Pacific
Send Feedback