Realtime Database Incident #17076

All services have returned to normal. Full incident report follows.

SUMMARY:

On Friday 25 August 2017, key Firebase infrastructure experienced intermittent outages, each about one hour long. This caused certain Firebase products to experience outages as well. We understand the pain that this caused developers that rely on Firebase, and apologize to everyone who was affected by the incident.

DETAILED DESCRIPTION OF IMPACT:

Incident 1: Started: 2017-08-25 11:19:00 PDT Incident Declared: 2017-08-25 11:50:00 PDT Ended: 2017-08-25 12:05:00 PDT

Incident 2 : Started: 2017-08-25 15:45:00 PDT Incident Declared: 2017-08-25 16:00:00 PDT Ended: 2017-08-25 16:50:00 PDT

During the outages, Firebase users could not create new Firebase projects, import projects from Google Cloud Platform, or deploy Cloud Functions. Additionally, developers could not access administrative functions and Firebase Console related to Firebase Realtime Database and Firebase Hosting. Customers using Legacy Firebase Auth also experienced a service interruption. Lastly, a small subset (<1%) of databases were inaccessible, with no data loss.

ROOT CAUSE:

A failed release deploy unexpectedly affected key Firebase infrastructure. The components were brought back online quickly. A second deployment was attempted after attempting to correct the problem, but this deployment was also faulty and resulted in a second interruption of service.

REMEDIATION AND PREVENTION: Once the outage was discovered, Google engineers moved critical infrastructure off of the affected servers to bring the service back online.

To prevent a future recurrence, Google engineers are: Making the Realtime Database infrastructure more robust by upgrading and isolating key infrastructure components. Improving the Realtime Database incident recovery process using lessons learned in this outage to reduce impact of future outages.

We apologize again for the inconvenience this issue caused our developers.

The issue has returned where some projects are unable to access their database. We will post an update with more information once it is available.

The issue has returned where some projects are unable to access their database. We will post an update with more information once it is available.

The issue has been resolved. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence.

Services should be coming back online now. We are waiting for further confirmation give the all-clear.

We are investigating the issue with Firebase Realtime Database where some projects are unable to access their database. We will post an update with more information once it is available.