Console Incident #17022
Issue with Firebase console
Incident began at 2017-08-25 16:26 and ended at 2017-08-25 16:49 (all times are US/Pacific).
|Aug 25, 2017||17:00||
On Friday 25 August 2017, key Firebase infrastructure experienced intermittent outages, each about one hour long. This caused certain Firebase products to experience outages as well. We understand the pain that this caused developers that rely on Firebase, and apologize to everyone who was affected by the incident.
DETAILED DESCRIPTION OF IMPACT:
Incident 1: Started: 2017-08-25 11:19:00 PDT Incident Declared: 2017-08-25 11:50:00 PDT Ended: 2017-08-25 12:05:00 PDT
Incident 2 : Started: 2017-08-25 15:45:00 PDT Incident Declared: 2017-08-25 16:00:00 PDT Ended: 2017-08-25 16:50:00 PDT
During the outages, Firebase users could not create new Firebase projects, import projects from Google Cloud Platform, or deploy Cloud Functions. Additionally, developers could not access administrative functions and Firebase Console related to Firebase Realtime Database and Firebase Hosting. Customers using Legacy Firebase Auth also experienced a service interruption. Lastly, a small subset (<1%) of databases were inaccessible, with no data loss.
A failed release deploy unexpectedly affected key Firebase infrastructure. The components were brought back online quickly. A second deployment was attempted after attempting to correct the problem, but this deployment was also faulty and resulted in a second interruption of service.
REMEDIATION AND PREVENTION: Once the outage was discovered, Google engineers moved critical infrastructure off of the affected servers to bring the service back online.
To prevent a future recurrence, Google engineers are: Making the Realtime Database infrastructure more robust by upgrading and isolating key infrastructure components. Improving the Realtime Database incident recovery process using lessons learned in this outage to reduce impact of future outages.
We apologize again for the inconvenience this issue caused our developers.
|Aug 25, 2017||16:59||
This issue is resolved now and the console should be back to normal for all projects which may have been affected. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence.
|Aug 25, 2017||16:26||
The issue has resurfaced where some projects may be unable to access their console. We will post an update with more information once it is available.