Error when mirroring data to BigQuery PartialFailureError

[zdenulo]

[REQUIRED] Step 2: Describe your configuration

• Extension name: Export Collections to BigQuery
• Extension version: 0.1.2

[REQUIRED] Step 3: Describe the problem

I set up the extension and upon inserting a document in Firestore, Cloud Function throws an error in logs:
Error when mirroring data to BigQuery { PartialFailureError
at request (/srv/node_modules/@google-cloud/bigquery/build/src/table.js:1548:23)
at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:141:9)
at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:422:22)
at onResponse (/srv/node_modules/retry-request/index.js:206:7)
at /srv/node_modules/teeny-request/build/src/index.js:222:13
at
at process._tickDomainCallback (internal/process/next_tick.js:229:7)
errors: [ { errors: [Array], row: [Object] } ],
name: 'PartialFailureError',
response:
{ kind: 'bigquery#tableDataInsertAllResponse',
insertErrors: [ [Object] ] },
message: '' }

And of course, there is no data in BigQuery table.

I saw there were similar errors in the recent past regarding batch import but since this is in Cloud Function, I'm creating a separate issue.

Cloud Function source location
https://storage.cloud.google.com/firebase-mod-sources-prod/e1a82bec6981c73a5fe6bff28483f3125ca703584c5fad43d1c4b7fa0f67d388

[matteosb]

I am seeing the same error for extensions installed in the last 24 hours. In some cases we get a permission denied error as well as the error above, even though the IAM roles look correct. Identical extensions installed day before yesterday are working fine:

{ Error: Access Denied: Dataset <project>:firestore_export: User does not have bigquery.datasets.get permission for dataset <project>:firestore_export.
    at Util.parseHttpRespBody (/srv/node_modules/@google-cloud/common/build/src/util.js:190:38)
    at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:134:117)
    at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:422:22)
    at onResponse (/srv/node_modules/retry-request/index.js:206:7)
    at /srv/node_modules/teeny-request/build/src/index.js:222:13
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
  code: 403,
  errors: 
   [ { message: 'Access Denied: Dataset <project>:firestore_export: User does not have bigquery.datasets.get permission for dataset <project>:firestore_export.',
       domain: 'global',
       reason: 'accessDenied' } ],
  response: undefined,
  message: 'Access Denied: Dataset <project>:firestore_export: User does not have bigquery.datasets.get permission for dataset <project>:firestore_export.' }```

[livingspree]

I'm facing the exact same issue as @zdenulo

Following this thread

[Shailevy]

Same here

[russellwheatley]

Hi @zdenulo, there was a fix for the firestore-bigquery-export that was merged into master (#192 updated). I suggest tracking the @firebaseextensions/firestore-bigquery-change-tracker package for its latest release which ought to fix the PartialFailureError you're currently experiencing.

[zdenulo]

@russellwheatley thanks for the info. Didn't you mean perhaps #192? I did my deployment via firebase web console. I now checked, i.e. downloaded code from the url I included (and based on which my Cloud Function is deployed) and CHANGELOG file there it says that it's release 0.1.2 which contains fixes for #192, #193 and #197. Maybe I'm missing something, Javascript/NodeJS/Typescript is exactly my area. on NPM latest release is 0.1.1

[pavitra15]

Same error here , tried but failed again.
Can anyone help??

[russellwheatley]

Sorry, you're correct, it is #192. I've updated my comment for anyone else coming to this thread. This is the package you ought to be tracking: https://www.npmjs.com/package/@firebaseextensions/firestore-bigquery-change-tracker. I would ignore the version number as it has been incorrectly set. This package is nested within the firestore-bigquery-export extension. Here is a reference to it: https://github.com/firebase/extensions/tree/next/firestore-bigquery-export/firestore-bigquery-change-tracker

[zdenulo]

@russellwheatley thanks for the clarification.

[balderdash]

@russellwheatley Thanks for looking into this bug. Do you have a rough idea of when a new version of @firebaseextensions/firestore-bigquery-change-tracker will be published?

[nedols]

I'm facing the exact same issue.
Following this thread

[pgoode41]

Same issue here. Following as well.

[daviduzan]

I'll join the followers