Crash Reporting Incident #16003

SUMMARY:

On September 12, a problem with Firebase Crash Reporting for Android (currently in beta) and the rollout of Google Play services 9.6.80 caused apps to hang on launch, resulting in an ANR (Application Not Responding) error dialog to be seen by end users. Once the error was detected, the rollout was immediately halted. On the morning of the 14th, we traced this to an interaction between Crash Reporting and Play services during the initialization of Crash Reporting. By September 17, a new build of Google Play services was made available to all affected devices that corrected the issue. We’re making changes to prevent recurrence of these issues in the future.

We know you're relying on us and that any outage, even affecting a small percentage of your end users, is unacceptable. We apologize to everyone who was affected by the incident.

DETAILED DESCRIPTION OF IMPACT:

Between 10 AM PDT on September 12 and 5 PM PDT on September 17, a small subset of devices triggered ANRs when launching apps that used Firebase Crash Reporting.

September 12 10:00 PDT: Google Play services 9.6.80 began a gradual rollout. September 13 21:00 PDT: First report of ANR from developers. September 14 8:00 PDT: Rollout of Google Play services 9.6.80 halted (approximately 3.7% of devices had installed the update by this point.) September 14 11:30 PDT: Root cause identified and fixed. September 15 13:30 PDT: Google Play services 9.6.83 made available to devices that had already received the 9.6.80 update. September 17 15:00 PDT: Approximately all affected devices have installed Google Play services 9.6.83, incident ends.

Two conditions were required in order for the ANR to manifest. First, the Android device needed to be running Google Play services 9.6.80 (rolled out to about 3.7% of all Android devices). Second, users needed to be opening an app with Firebase Crash Reporting enabled, that had not been opened within the prior week, or had been just installed.

Any app/device pair that met these conditions would fail to operate successfully, until an updated version of Google Play services was automatically installed on affected devices, or Firebase Crash Reporting was disabled in the APK by the developer.

ROOT CAUSE:

The root cause was due to a rare and unexpected behavior in the Android platform. Additionally, there was a design flaw in the code, and changes made that led to a bug in the build.

Neither automated nor manual testing covered the functionality which was broken, allowing it to slip through. Internal release of the Google Play services build to a sizeable test pool of Google devices, prior to external release, also failed to catch the break.

REMEDIATION AND PREVENTION:

An updated version of Google Play services (9.6.83) has been made available to all affected users, which corrects the underlying behavior that triggered the ANR. Please bear in mind that this fix depends on the device’s ability to download and install the update, so devices that are low on disk space or have an unreliable network connection may not see the correction until the update is performed.

To prevent future occurrence, Google engineers are:

1) Increasing the size of the suite of internal apps included in automated testing.

2) Adding monitoring of several core functions of Play services to alert us of similar classes of issues more quickly.

3) Changing the implementation of several core functions to entirely avoid the deadlock that caused the ANR.

We are very appreciative of the rapid and accurate feedback we received from external sources that allowed us to diagnose and fix the problem relatively quickly. To those of you who reached out, thank you for your alertness.

We've fixed the issue with Firebase Crash initialization in Google Play Services 9.6.83. Users should receive the update sometime within the next two days. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence.

We’re testing a potential fix for the issue. More information will be posted once we have confirmed the results. All affected customers should expect a new version of Google Play services (9.6.83) by 9/17.

We are investigating an issue with Firebase Crash initialization for users who have updated to Google Play services 9.6.80. We will provide more information by 11:42 US/Pacific on September 15th.