Realtime Database Incident #18001

Investigating an issue with Realtime Database.

Incident began at 2018-01-09 14:30 and ended at 2018-01-09 22:40 (all times are US/Pacific).

Date Time Description
Jan 09, 2018 23:30

The issue has been solved at 10:40 PM US/Pacific.

We found an infrastructure network misconfiguration that is suspected to be the culprit. We'll perform investigation in the coming days to confirm that and figure out ways to mitigate in the future.

Jan 09, 2018 19:24

The system has again recovered at around 6:50 PM US/Pacific. However, the increased latency is being encountered intermittently and the team is still monitoring future recurrences.

Jan 09, 2018 18:41

The issue has been encountered again around 6:20 PM US/Pacific. We will provide another status update as soon as possible.

Jan 09, 2018 18:16

The system has recovered at around 5:30 PM US/Pacific, but we're still looking into the culprit and it may recur at any time. We'll continue to monitor and provide any updates.

Jan 09, 2018 17:48

We're observing increased latency that started around 2:30 PM US/Pacific but intensified around 4:25 PM US/Pacific that affects all customers. Some instances with high load may observe timeouts and unavailability.

We will provide more information as soon as possible.

All times are US/Pacific
Send Feedback