Cloud Messaging Incident #18014

FCM high latency issue

Incident began at 2018-08-31 04:38 and ended at 2018-08-31 12:20 (all times are US/Pacific).

Date Time Description
Oct 17, 2018 10:25

SUMMARY

FCM message senders (using both XMPP and HTTP protocols) experienced abnormally high error rates and latency when communicating with FCM servers for sends and upstream messages. Other interactions with FCM, such as device and topic registration, were unaffected.

DETAILED DESCRIPTION OF IMPACT

Low % impacted (<0.1% of traffic): 29 August 2018 - 19 September 2018

High impact (at least 0.1% of traffic, with peaks to about 8% of traffic): 2018-08-30 15:30 through 2018-08-31 11:00 2018-09-06 17:30 through 2018-09-07 21:30 2018-09-11 17:15 through 2018-09-14 21:00

ROOT CAUSE

Recent updates to the FCM software stack caused unexpectedly high server resource usage. That, combined with a bug that caused server load to randomly spike, created consistently poor performance for a small but significant percentage of FCM traffic. Due to the random nature of the spikes, it took longer than usual for FCM engineers to diagnose the issue.

REMEDIATION AND PREVENTION

During the first two of the three high impact periods, FCM engineers were able to relieve the effects to a substantial degree by providing temporary extra capacity to the FCM services. Since then, FCM servers have been granted permanent additional compute capacity.

The third period of high impact was caused by a high proportion of requests of a specific type that were likely to trigger the bug that caused server resource usage to spike. FCM engineers isolated these problematic requests into dedicated server pools (completed about 2018-09-14 14:00), and then identified and resolved the bug (completed about 2018-09-14 20:00).

FCM engineers are pursuing preventative measures, such as improved load management and balancing, to make sure this is less likely to occur in the future.

Aug 31, 2018 12:21

The latency issue with FCM should have been resolved for all affected users as of 12:20 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence.

Aug 31, 2018 10:00

The latency issue with FCM should have been resolved for a majority of the users, and we expect a full resolution in the near future. We will provide another status update as soon as possible.

Aug 31, 2018 07:50

We're working on mitigating the latency issues. We will provide another status update by 10:00 AM US/Pacific with current details.

Aug 31, 2018 06:53

We are experiencing a high latency issue with FCM delivery of messages beginning at 2018-08-31 04:38 US/Pacific.

For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 07:50 US/Pacific with current details.

All times are US/Pacific
Send Feedback