Firebase Status Dashboard
NOTE
For incidents related to Cloud Functions, Cloud Firestore and Cloud Storage, please see Cloud Status Dashboard. And for incidents related to Google Analytics, please see Ads Status Dashboard.
Incident affecting Cloud Messaging
Errors delivering to a small subset of registrations
Incident began at 2024-10-14 15:30 and ended at 2024-10-18 15:25 (all times are US/Pacific).
Date | Time | Description | |
---|---|---|---|
| 18 Oct 2024 | 15:29 PDT | Our mitigations have landed and this issue is now resolved. |
| 18 Oct 2024 | 14:59 PDT | We are aware of an issue affecting less than 1% of registrations, which causes message delivery to fail with Internal Server Errors. The issue remains fixed, not affecting any new registrations. Previously affected registrations are slowly recovering. We are working on a fix that will accelerate the recovery for the remaining affected registrations. We are expecting some recovery within the next 24 hours and aiming for full recovery by early next week. If you are experiencing Internal error rates or 500s greater than 1%, consider altering your retry behavior to avoid error amplification. We will provide an update on Oct 19 3pm PST. |
| 17 Oct 2024 | 15:00 PDT | We are aware of an issue affecting less than 1% of registrations, which causes message delivery to fail with Internal Server Errors. The issue remains fixed, not affecting any new registrations. Previously affected registrations are slowly recovering. We are working on a fix that will accelerate the recovery for the remaining affected registrations. If you are noticing Internal error rates greater than 1%, consider altering your retry behavior to avoid error amplification. We will provide an update on Oct 18 3pm PST. |
| 16 Oct 2024 | 14:20 PDT | We are aware of an issue affecting less than 1% of registrations, which causes message delivery to fail with Internal Server Errors. The issue is understood and mitigated. The system is slowly recovering, and we are working to expedite recovery. We continue to monitor the situation and will provide an update on Oct 17 3pm PST. |
| 15 Oct 2024 | 08:38 PDT | We are aware of an issue affecting a small number of registrations causing delivery to fail on a subset of the traffic. The issue is understood and mitigated. The system is slowly self healing and we are conducting additional restoration work which will take a few days. We continue to monitor the situation and will provide an update on Oct 16 10am PST |
| 14 Oct 2024 | 19:18 PDT | We believe we have mitigated issues with the Send APIs and have observed an automatic gradual recovery. We will continue to monitor the situation and provide an update at 2:30 AM ET. |
| 14 Oct 2024 | 18:27 PDT | We believe we have mitigated issues with the Register APIs, the Send APIs remain impacted. We are continuing to investigate the problem and will provide an update at 10:30 PM ET. |
| 14 Oct 2024 | 17:25 PDT | We're aware of a problem with Firebase Cloud Messaging affecting a subset of traffic to Send & Register APIs. We will provide an update by 9:30PM ET detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change. |
- All times are US/Pacific