Cloud Functions Incident #17017
Investigating an issue with Fuctions
Incident began at 2017-08-25 16:33 and ended at 2017-08-25 16:50 (all times are US/Pacific).
|Aug 25, 2017||16:58||
The issue has been resolved. A full incident report follows:
On Friday 25 August 2017, key Firebase infrastructure experienced intermittent outages, each about one hour long. This caused certain Firebase products to experience outages as well. We understand the pain that this caused developers that rely on Firebase, and apologize to everyone who was affected by the incident.
DETAILED DESCRIPTION OF IMPACT:
Incident 1: Started: 2017-08-25 11:19:00 PDT Incident Declared: 2017-08-25 11:50:00 PDT Ended: 2017-08-25 12:05:00 PDT
Incident 2 : Started: 2017-08-25 15:45:00 PDT Incident Declared: 2017-08-25 16:00:00 PDT Ended: 2017-08-25 16:50:00 PDT
During the outages, Firebase users could not create new Firebase projects, import projects from Google Cloud Platform, or deploy Cloud Functions. Additionally, developers could not access administrative functions and Firebase Console related to Firebase Realtime Database and Firebase Hosting. Customers using Legacy Firebase Auth also experienced a service interruption. Lastly, a small subset (<1%) of databases were inaccessible, with no data loss.
A failed release deploy unexpectedly affected key Firebase infrastructure. The components were brought back online quickly. A second deployment was attempted after attempting to correct the problem, but this deployment was also faulty and resulted in a second interruption of service.
REMEDIATION AND PREVENTION: Once the outage was discovered, Google engineers moved critical infrastructure off of the affected servers to bring the service back online.
To prevent a future recurrence, Google engineers are: Making the Realtime Database infrastructure more robust by upgrading and isolating key infrastructure components. Improving the Realtime Database incident recovery process using lessons learned in this outage to reduce impact of future outages.
We apologize again for the inconvenience this issue caused our developers.
|Aug 25, 2017||16:33||
The issue we experienced earlier today has resurfaced which may be causing problems with Function deployment. We will post an update with more information once it is available.