Realtime Database Incident #17077
Issue with Firebase Realtime Database
Incident began at 2017-08-25 16:01 and ended at 2017-08-25 17:00 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| Aug 25, 2017 | 17:01 | SUMMARY: On Friday 25 August 2017, key Firebase infrastructure experienced intermittent outages, each about one hour long. This caused certain Firebase products to experience outages as well. We understand the pain that this caused developers that rely on Firebase, and apologize to everyone who was affected by the incident. DETAILED DESCRIPTION OF IMPACT: Incident 1: Started: 2017-08-25 11:19:00 PDT Incident Declared: 2017-08-25 11:50:00 PDT Ended: 2017-08-25 12:05:00 PDT Incident 2 : Started: 2017-08-25 15:45:00 PDT Incident Declared: 2017-08-25 16:00:00 PDT Ended: 2017-08-25 16:50:00 PDT During the outages, Firebase users could not create new Firebase projects, import projects from Google Cloud Platform, or deploy Cloud Functions. Additionally, developers could not access administrative functions and Firebase Console related to Firebase Realtime Database and Firebase Hosting. Customers using Legacy Firebase Auth also experienced a service interruption. Lastly, a small subset (<1%) of databases were inaccessible, with no data loss. ROOT CAUSE: A failed release deploy unexpectedly affected key Firebase infrastructure. The components were brought back online quickly. A second deployment was attempted after attempting to correct the problem, but this deployment was also faulty and resulted in a second interruption of service. REMEDIATION AND PREVENTION: Once the outage was discovered, Google engineers moved critical infrastructure off of the affected servers to bring the service back online. To prevent a future recurrence, Google engineers are: Making the Realtime Database infrastructure more robust by upgrading and isolating key infrastructure components. Improving the Realtime Database incident recovery process using lessons learned in this outage to reduce impact of future outages. We apologize again for the inconvenience this issue caused our developers and your end users! |
|
| Aug 25, 2017 | 17:00 | Services are back online now. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence. |
|
| Aug 25, 2017 | 16:01 | The issue has resurfaced where some projects are unable to access their database. We will post an update with more information once it is available. |