Firebase Status Dashboard

Firebase Status Dashboard
Incidents
Issue with Realtime Database: REST and Functions are timing out for some projects

This page provides status information on the services that are part of Firebase. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://firebase.google.com/.

NOTE

For incidents related to Cloud Functions, Cloud Firestore and Cloud Storage, please see Cloud Status Dashboard. And for incidents related to Google Analytics, please see Ads Status Dashboard.

Available
Service information
Service disruption
Service outage

Incident affecting Realtime Database

Issue with Realtime Database: REST and Functions are timing out for some projects

Incident began at 2018-06-26 02:10 and ended at 2018-06-26 04:58 (all times are US/Pacific).

	Date	Time	Description
	6 Jul 2018	12:00 PDT	SUMMARY: Starting at 2:10 AM PDT on Tuesday 26 June, 2018, some developers using the Firebase Realtime Database experienced timeouts for about 3 hours, until 4:58 AM PDT of the same day. DETAILED DESCRIPTION OF IMPACT: REST API and Admin SDK requests to Realtime Database instances hosted on NSS-212, 230, and 516 timed out due to an internal networking bug that affected authorization checks for access tokens used by service accounts and project owners. We estimate that 5% of projects were affected by this bug. ROOT CAUSE: The networking bug caused an exhaustion of socket connections to the internal authorization service that validates access tokens. Eventually, there were no open sockets left to make requests, and developers began to experience timeouts. The issue was identified by external developers that reached out to support, who then paged the Realtime Database oncall. REMEDIATION AND PREVENTION: The initial, short-term fix was to drain and restart the servers to reset their connections. This unblocked the authentication requests and stopped the timeouts. To prevent this from happening in the future, Realtime Database engineers deployed a fix for the networking issue on Wednesday 27 June, 2018 and added alerts to notify the Realtime Database oncall of a similar issue in order to minimize the time in which external developers experience timeouts.
	26 Jun 2018	05:13 PDT	We experienced an issue with Realtime Database where REST and Functions had timeouts for some projects. The issue started at 2:10 AM US/Pacific and has been resolved for all affected users as of 4:58 AM US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence. Apologies for the inconvenience.

Date

Time

Description

6 Jul 2018

12:00 PDT

SUMMARY:

Starting at 2:10 AM PDT on Tuesday 26 June, 2018, some developers using the Firebase Realtime Database experienced timeouts for about 3 hours, until 4:58 AM PDT of the same day.

DETAILED DESCRIPTION OF IMPACT:

REST API and Admin SDK requests to Realtime Database instances hosted on NSS-212, 230, and 516 timed out due to an internal networking bug that affected authorization checks for access tokens used by service accounts and project owners. We estimate that 5% of projects were affected by this bug.

ROOT CAUSE:

The networking bug caused an exhaustion of socket connections to the internal authorization service that validates access tokens. Eventually, there were no open sockets left to make requests, and developers began to experience timeouts. The issue was identified by external developers that reached out to support, who then paged the Realtime Database oncall.

REMEDIATION AND PREVENTION:

The initial, short-term fix was to drain and restart the servers to reset their connections. This unblocked the authentication requests and stopped the timeouts.

To prevent this from happening in the future, Realtime Database engineers deployed a fix for the networking issue on Wednesday 27 June, 2018 and added alerts to notify the Realtime Database oncall of a similar issue in order to minimize the time in which external developers experience timeouts.

26 Jun 2018

05:13 PDT

We experienced an issue with Realtime Database where REST and Functions had timeouts for some projects. The issue started at 2:10 AM US/Pacific and has been resolved for all affected users as of 4:58 AM US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence. Apologies for the inconvenience.

All times are US/Pacific