Test Lab Incident #19010
We are investigating increased latency with Cloud Firebase Test Lab
Incident began at 2019-06-25 10:53 and ended at 2019-06-25 19:31 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| Jun 25, 2019 | 19:31 | The issue of increased latency for tests in Firebase Test Lab has been resolved for all affected projects as of Tuesday, 2019-06-25 19:20 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. |
|
| Jun 25, 2019 | 18:32 | The mitigation steps are progressing well and we will continue to monitor test latency closely. We will provide another status update by Tuesday, 2019-06-25 19:30 US/Pacific with current details. |
|
| Jun 25, 2019 | 17:38 | Mitigation is progressing but affected Projects will continue to see increased latency for virtual devices. We will provide another status update by Tuesday, 2019-06-25 18:30 US/Pacific with current details. |
|
| Jun 25, 2019 | 15:42 | We are still monitoring Firebase Test Lab projects impacted by increased testing latency. We have fully mitigated this issue for physical devices, but it still remains for virtual devices. We will provide another status update by Tuesday, 2019-06-25 17:30 US/Pacific with current details. |
|
| Jun 25, 2019 | 14:13 | We are still monitoring projects impacted by increased testing latency with Firebase Test Lab to ensure that this issue does not recur. The issue should be resolved for the majority of customers. We will provide another status update by Tuesday, 2019-06-25 15:10 US/Pacific with current details. |
|
| Jun 25, 2019 | 13:18 | The issue with Firebase Test Lab should be resolved for the majority of projects and we expect a full resolution in the near future. We will provide another status update by Tuesday, 2019-06-25 14:06 US/Pacific with current details. |
|
| Jun 25, 2019 | 12:33 | The Firebase Test Lab service is continuing to experience increased latency. Our engineering team estimates a mitigation within ~3 hours. We will provide another status update by Tuesday, 2019-06-25 15:30 US/Pacific with details. Earlier updates may be made if more information is discovered and relevant. |
|
| Jun 25, 2019 | 11:58 | The Firebase Test Lab service is experiencing increased latency. Our Engineering Team is continuing the investigation and believe to have found root cause. We are still working on mitigating this issue and we will provide another status update by Tuesday, 2019-06-25 12:28 US/Pacific with current details. |
|
| Jun 25, 2019 | 11:09 | The Firebase Test Lab service is experiencing increased latency on responses. We will provide another status update by Tuesday, 2019-06-25 11:45 US/Pacific with current details. |