Cloud Messaging Incident #17001
Issue with Firebase Notifications delay in calculating sent counts
Incident began at 2017-02-28 08:30 and ended at 2017-03-03 08:00 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| Mar 03, 2017 | 08:08 | The issue with delays in Firebase Notifications calculating sent counts should have been resolved for all affected users as of 07:23 AM US/Pacific. The backfill has been completed and we believe all affected campaigns should now have the correct sent counts. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence. |
|
| Mar 03, 2017 | 02:13 | As of Friday, 2017-03-03 12:30 AM US/Pacific, the sent count processing is working again for new messages. This is due to the effect of the roll back and is the time that we expected. This leaves messages sent between Tuesday, 2017-02-28 08:30 AM US/Pacific and end of Wednesday, 2017-03-01 still without sent counts. We are actively working to back fill the sent counts for those messages but new messages should not be affected. |
|
| Mar 01, 2017 | 11:07 | We are applying a second solution and that is expected to result in new messages receiving their sent counts some time on Friday 3 March. Messages that have lost their sent count will subsequently be back filled. |
|
| Mar 01, 2017 | 08:25 | The solution hasn't propagated through and we are looking at alternatives. We will provide another status update by 11:25 AM US/Pacific with current details. |
|
| Mar 01, 2017 | 03:25 | We now know the cause and a solution is underway but will take several hours (could be 5 or more, could be less) to take affect. We will provide another status update by 08:25 AM US/Pacific with current details. |
|
| Mar 01, 2017 | 02:25 | We are still investigating the issue with Firebase Notifications delay in calculating sent counts. Current data indicates that all users are affected by this issue. We will provide another status update by 03:25 AM US/Pacific with current details. |
|
| Mar 01, 2017 | 01:26 | We are experiencing delays in Firebase Notifications calculating sent counts beginning at Tuesday, 2017-02-28 08:30 AM US/Pacific. Current data indicates that all users are affected by this issue. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 02:25 AM US/Pacific with current details. |