Console Incident #19014

The issue with Firebase Console project and app creation failures, along with failures to enable new APIs has been resolved for all affected projects as of Thursday, 2019-07-25 18:24 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

The issue with Firebase Console project and app creation failures, along with failures to enable new APIs should be resolved for the majority of projects. Currently, approximately 5% of the requests are still failing. Our engineers are still working to mitigate the issue and we expect a full resolution in the near future. We will provide another status update by Thursday, 2019-07-25 18:00 US/Pacific with current details.

The rate of errors is decreasing. We will provide another status update by Thursday, 2019-07-25 16:45 US/Pacific with current details.

We are still seeing errors. Our Engineers have implemented a mitigation and are now monitoring the situation before gradually rolling it out to a wider audience. We will provide another status update by Thursday, 2019-07-25 15:45 US/Pacific with current details.

We are still seeing errors. Our Engineers are attempting several mitigation strategies in parallel and are beginning to see positive results. We will provide another status update by Thursday, 2019-07-25 14:45 US/Pacific with current details.

We are investigating errors when creating a new project or application or when enabling new APIs and services on existing projects. Mitigation work is currently still underway by our Engineering Team. We will provide another status update by Thursday, 2019-07-25 13:45 US/Pacific with current details.

We are investigating errors when creating a new project or application or when enabling new APIs and services on existing projects. Our engineering team has identified a root cause, and mitigation work is currently underway. Efforts to accelerate recovery using the previous mitigation were not effective and the issue has been escalated internally, we are continuing to work on a new mitigation. We will provide another status update by Thursday, 2019-07-25 12:30 US/Pacific with current details.

We are investigating errors when creating a new project or application or when enabling new APIs and services on existing projects. Our engineering team has identified a root cause, and mitigation work is currently underway. Mitigation appears effective, and we are working to accelerate recovery. We will provide another status update by Thursday, 2019-07-25 10:30 US/Pacific with current details.

We are investigating errors when creating a new project or application or when enabling new APIs and services on existing projects. Our engineering team has identified a root cause, and mitigation work is currently underway. Mitigation appears effective, and we are working to accelerate recovery. We will provide another status update by Thursday, 2019-07-25 09:15 US/Pacific with current details.

We are investigating errors when creating a new project or application or when enabling new APIs and services on existing projects. Our Engineering Team believes they have identified the root cause and mitigation work is underway. We will provide another status update by Thursday, 2019-07-25 09:15 US/Pacific with current details.

We are investigating errors when creating a new project or application or when enabling new APIs and services on existing projects. Our Engineering Team is investigating possible causes. We will provide another status update by Thursday, 2019-07-25 08:15 US/Pacific with current details.