Cloud Functions Incident #18021

SUMMARY: A change to Cloud Functions for Firebase caused a group of Firebase developers to run out of build-time quota, since builds of their functions took significantly longer than previously.

DETAILED DESCRIPTION OF IMPACT: A group of Firebase developers that have functions with many dependencies, in particular developers running continuous integration (CI) builds, ran out of build-time quota, since affected builds took up to 200 times longer than they had before.

ROOT CAUSE: A change, rolled out on 2018-04-18, switched to a suboptimal caching strategy that increased build time for functions deployed with more than 20 direct dependent npm modules (found in the “dependencies” section of the package.json). This caused individual deployments to use more build time quota, which meant that functions developers with CI builds quickly ate through their quota.

REMEDIATION AND PREVENTION: The immediate, short-term fix was to temporarily lift quotas on affected projects to unblock their builds.

Since the incident, the offending change has been reverted. Additionally, as part of the postmortem process, improvements to automated monitoring have been identified to more quickly detect and mitigate a similar issue in the future.

The issue with the deployment quotas on Cloud Functions should have been resolved for all affected projects as of 07:42pm US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence.

A recent change in Cloud Functions may cause increased deployment times (build times) for functions that have many dependencies, because caching for this case has been reduced. The increased build time causes higher usage of the build-time quota. For Firebase, this problem may be more frequent because it is common to have all functions, with all their dependencies in the same source file.

To reduce the build time, Firebase developers can remove unnecessary dependencies (if there are such). Deploying functions together with node_modules directory may help too.

We're working on reverting this change.

We are still investigating the issue with Cloud Functions for Firebase regarding deployment quotas. If you are experiencing an issue, please contact support (https://firebase.google.com/support/).

We will provide another status update by 8:30pm US/Pacific.

We are still investigating the issue with Cloud Functions for Firebase regarding deployment quotas. If you are experiencing an issue, please contact support (https://firebase.google.com/support/).

We will provide another status update by 4:30pm US/Pacific.

We are still investigating the issue with Cloud Functions for Firebase regarding deployment quotas. We will provide another status update by 4:00pm US/Pacific.

We are investigating an issue related to deployment quotas for Cloud Functions for Firebase. We will provide more information by 3:30pm US/Pacific.