🐛 [Stream Firestore to BigQuery] Update cloud function extension to gen 2

[leslie-zi-pan]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository.
If you have a general question, need help debugging, or fall into some
other category use one of these other channels:

• For general technical questions, post a question on StackOverflow
  with the firebase tag.
• For general Firebase discussion, use the firebase-talk
  google group.
• To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to
  Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

• Extension name: Stream Firestore to BigQuery
• Extension version: 0.1.49

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

Failed extension deployment - specifically the firestore component of this extension. The extension allows database targetting (other than the default DB). However, deployment of this extension is still in 1st Gen Cloud function which limits the trigger to default database. This is contradictory to what the extension feature allows as there is a field to specify database ID. Please update to Gen 2. Thank you.

[mikeymace]

Hi!

Do we have a time frame for when the extensions will be upgraded to use the 2nd Gen Cloud Functions?

We are currently in the process of regionalising our Firestore databases (i.e. eur, us, shared) and we currently rely on the 'Stream Firestore to BigQuery' extension to get our data into BigQuery. Ideally, each database would be mapped to an equivalent (localised) dataset in BigQuery.

I have found the following behaviour for firebase/firestore-bigquery-export@0.1.49:

• If I don't set the DATABASE_ID .env variable then the Cloud Function automatically defaults to 1st gen and DATABASE_ID = (default) see screenshot below.
• If I do try and set DATABASE_ID = <location> in the .env then I get the following error message during deployment: ResourceErrorCode":"400","ResourceErrorMessage":"Cannot create Firestore trigger for .../databases/us-west2. Triggers must be created with a \"(default)\" Firestore Native database. See https://firebase.google.com/docs/firestore/extend-with-functions#limitations.

We also rely on npx firebaseextensions/fs-bq-import-collection for manual backfilling of existing Firestore data. Can you also confirm that this will be updated so that we can pass the database_id as an argument?

This is urgent from our side as our database migration is planned to go ahead early June and therefore any information on expected timeframes for this to be upgraded would be greatly appreciated otherwise unfortunately we will be forced to look for an alternative.

Thank you!

[IchordeDionysos]

@mikeymace I'd suggest looking for an alternative, e.g. copy the code and deploy it directly (aka not as Firebase Extension).
This will take some time until it's fixed ...

[pr-Mais]

Hello everyone, tagging this as a feature request.

Unfortunately, it's not possible now to upgrade any extension to v2 Firestore triggers since extensions don't yet support them, once it does we will update the triggers.

For the contradiction between the params and actual functionality, I'm tracking this as an issue here #2090.

[basvandorst]

+1

general note: would be great if all (official) Firebase extensions also moving towards gen 2

[adonistseriotis]

Hello everyone, tagging this as a feature request.

Unfortunately, it's not possible now to upgrade any extension to v2 Firestore triggers since extensions don't yet support them, once it does we will update the triggers.

For the contradiction between the params and actual functionality, I'm tracking this as an issue here #2090.

I think somehow Algolia has implemented something similar on their Extension. Maybe, it is possible to do this here?

Default configuration is hardcoded here and even if this fails due to taskQueues implementation, the approach could be changed to match Algolia's sync approach. What do you think @pr-Mais ?

[tlloydukdev]

Default configuration is hardcoded here and even if this fails due to taskQueues implementation, the approach could be changed to match Algolia's sync approach. What do you think @pr-Mais ?

+1 from me

[rummanext]

+1 from me

[cabljac]

I will escalate this to the firebase team, but from what I understand when we tried to do this with a similar approach to Algolia, we ran into the issue -

#2090 (comment)

If anyone has tested that Algolia extension with a non-default firestore instance, please let me know