Test Lab Incident #19008
We are investigating an issue with Firebase Test Lab tests entering a pending state.
Incident began at 2019-03-25 14:43 and ended at 2019-03-25 18:44 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| Mar 25, 2019 | 18:44 | The issue with Firebase Test Lab tests entering a 'pending' state has been resolved for all affected projects as of Monday 2019-03-25 18:40 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. |
|
| Mar 25, 2019 | 16:26 | The rate of tests stuck in a 'pending' state is decreasing. iOS and Android Virtual tests have processed the vast majority of their backlogs. Currently, only Android Physical tests still remain delayed. Google Engineers are working to mitigate the issue. We will provide another status update by Monday, 2019-03-25 18:00 US/Pacific with current details. |
|
| Mar 25, 2019 | 15:25 | We are currently experiencing an issue with Firebase Test Lab tests entering a 'pending' state beginning at Monday, 2019-03-25 14:21 US/Pacific. Correcting a previous statement, it seems all test types are still impacted by the issue, with Android Virtual tests being the most impacted. Some tests have resumed and Google Engineers are currently investigating potential root causes. For everyone who is affected, we apologize for the disruption. We will provide an update by Monday, 2019-03-25 16:30 US/Pacific with current details. |
|
| Mar 25, 2019 | 15:06 | We are currently experiencing an issue with Firebase Test Lab tests entering a pending state beginning at Monday, 2019-03-25 14:21 US/Pacific. Android Virtual, Android Physical, and iOS tests were all impacted. Currently, Android Physical and iOS tests have returned to normal operation. The backlog of Android Virtual tests is still being processed. Google Engineers are currently investigating potential root causes. For everyone who is affected, we apologize for the disruption. We will provide an update by Monday, 2019-03-25 16:00 US/Pacific with current details. |
|
| Mar 25, 2019 | 14:43 | We are currently experiencing an issue with Firebase Test Lab beginning at Monday, 2019-03-25 14:21 US/Pacific. Early investigation indicates that all customers are currently affected by this issue. Impacted users will see that their jobs have entered into a pending state and are queued to run. For everyone who is affected, we apologize for the disruption. We will provide an update by Monday, 2019-03-25 15:30 US/Pacific with current details. |
|
| Mar 25, 2019 | 14:43 | We are investigating an issue with Firebase Test Lab tests entering a pending state. |