Authentication Incident #17001

SUMMARY:

On Monday 27 March 2017, sign in with Facebook stopped working for Firebase Auth web users for 14 hours. We understand the pain that this caused developers who rely on Facebook to sign users into their apps, and apologize to everyone who was affected by the incident.

DETAILED DESCRIPTION OF IMPACT:

On Monday 27 March 2017 from 10:00am PDT until 12:00am PDT on Tuesday 28 March 2017, sign in with Facebook failed for Firebase Auth when used in web redirection or popup mode. The Firebase Auth backend returned http 400 "A system error has occurred".

Android and iOS clients were not affected.

ROOT CAUSE:

Facebook decommissioned Graph API V2.2 around 10:00 am Mar. 27th, 2017 and changed the oauth2 token response format from url-encode to json, which broke the response parser in the Firebase Auth backend. Firebase Android/iOS apps were not affected because they use a different Facebook API to verify their access token, which remained unchanged.

Our internal error reporting categorized the error incorrectly, meaning that we only found out about this due to developers reporting it on our support channels.

REMEDIATION AND PREVENTION:

When the issue was identified as caused by Facebook’s deprecation of Graph API v2.2, Firebase Auth engineers updated our systems to be compatible with Graph API v2.3.

To prevent a future recurrence, Google engineers are: Setting recurring reminders based on Facebook’s Graph API versioning and deprecation schedule to ensure proactive updates in the future. Improving monitoring for Facebook auth traffic so that our systems catch issues like this before external developers do.

We apologize again for the inconvenience this issue caused our customers.

The issue has been resolved. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence.

Service has partially recovered, to approximately 20% of affected users. We will provide an update within 2 hours.

We experienced an issue with Facebook Authentication. Android/iOS apps are not affected. Web apps using signInWithPopup or signInWithRedirect would not login with Facebook. We identified the root cause and are working on a fix. We will provide an update as soon as possible.

We are investigating an issue with Firebase Authentication where developers are seeing 400 errors with Facebook Login. We will provide an update as soon as we have more information.