Firebase Status Dashboard

This page provides status information on the services that are part of Firebase. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://firebase.google.com/.

NOTE

For incidents related to Cloud Functions, Cloud Firestore and Cloud Storage, please see Cloud Service Health. For incidents related to Google Analytics, please see Ads Status Dashboard.

Incident affecting Cloud Messaging

FCM high latency issue

Incident began at 2018-08-31 04:38 and ended at 2018-08-31 12:20 (all times are US/Pacific).

Date Time Description
17 Oct 2018 10:24 PDT

SUMMARY

FCM message senders (using both XMPP and HTTP protocols) experienced abnormally high error rates and latency when communicating with FCM servers for sends and upstream messages. Other interactions with FCM, such as device and topic registration, were unaffected.

DETAILED DESCRIPTION OF IMPACT

Low % impacted (<0.1% of traffic): 29 August 2018 - 19 September 2018

High impact (at least 0.1% of traffic, with peaks to about 8% of traffic): 2018-08-30 15:30 through 2018-08-31 11:00 2018-09-06 17:30 through 2018-09-07 21:30 2018-09-11 17:15 through 2018-09-14 21:00

ROOT CAUSE

Recent updates to the FCM software stack caused unexpectedly high server resource usage. That, combined with a bug that caused server load to randomly spike, created consistently poor performance for a small but significant percentage of FCM traffic. Due to the random nature of the spikes, it took longer than usual for FCM engineers to diagnose the issue.

REMEDIATION AND PREVENTION

During the first two of the three high impact periods, FCM engineers were able to relieve the effects to a substantial degree by providing temporary extra capacity to the FCM services. Since then, FCM servers have been granted permanent additional compute capacity.

The third period of high impact was caused by a high proportion of requests of a specific type that were likely to trigger the bug that caused server resource usage to spike. FCM engineers isolated these problematic requests into dedicated server pools (completed about 2018-09-14 14:00), and then identified and resolved the bug (completed about 2018-09-14 20:00).

FCM engineers are pursuing preventative measures, such as improved load management and balancing, to make sure this is less likely to occur in the future.
31 Aug 2018 12:21 PDT

The latency issue with FCM should have been resolved for all affected users as of 12:20 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence.
31 Aug 2018 10:05 PDT

The latency issue with FCM should have been resolved for a majority of the users, and we expect a full resolution in the near future. We will provide another status update as soon as possible.
31 Aug 2018 07:50 PDT

We're working on mitigating the latency issues. We will provide another status update by 10:00 AM US/Pacific with current details.
31 Aug 2018 06:53 PDT

We are experiencing a high latency issue with FCM delivery of messages beginning at 2018-08-31 04:38 US/Pacific.

For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 07:50 US/Pacific with current details.