Segregate table profile extraction of bigquery to its own extractor

[mabdh]

Is your feature request related to a problem? Please describe.
In our existing bigquery extractor, we collect both bigquery table metadata & its profile (common join & filter) based on audit log. But since the data from audit log are not always having a 1-1 mapping to the table metadata (Sometimes audit log data has out of project context data), we always need to build a map of table stats from the audit log data and later we match the metadata to the map based on table urn.
The approach consumed high memory and can caused OOM error.

Describe the solution you'd like
Since our current compass support upsert-patch asset update, we might could decouple bigquery metadata extraction and bigquery table profile extraction to reduce the memory-footprint of a job.

[mabdh]

@ravisuhag @StewartJingga @AkarshSatija @scortier @GrayFlash any thought on this?

[ravisuhag]

Yes, Either decouple or we need to think of multiple mode support in a single extractor.

[StewartJingga]

Another issue is, for each extractor, we are fetching/processing the same profile data since extractors cannot share information even when they are in one single run.

One idea would be for meteor to have another hook to:

• triggered once in every run
• run before initiating/running a recipe
• can store data that can be accessed to all recipes in that run only

Later, this proposed hook can also help us when trying to populate user information to the metadata.

For example, it will allow us to initially fetch all shield users first, then use that shield information to map user's email to shield's uuid as opposed to each extractor making the same call over and over again.

wdyt? @ravisuhag @mabdh @scortier @GrayFlash

[ravisuhag]

@StewartJingga My worry is that it might make the interface a bit more complicated.

[mabdh]

@StewartJingga @ravisuhag
An idea that is less destructive is:

We are currently getting log with this filter query

protoPayload.methodName="jobservice.jobcompleted" AND ` +
	`resource.type="bigquery_resource" AND NOT ` +
	`protoPayload.serviceData.jobCompletedEvent.job.jobConfiguration.query.query:(INFORMATION_SCHEMA OR __TABLES__) AND ` +
	`timestamp >= "%s" AND timestamp < "%s"

collecting all table stats in a hash map and then use the mapping to assign it to the table fetched.

collect table stats for a specific time period and store to hash map > run extractor to extract table from bq > get a table > map table fetched from bq to table profile > build table metadata > sink

---

Instead of doing that, we can reduce the hash map for all data within time period by adding filter table name. The filter would be

protoPayload.methodName="jobservice.jobcompleted" AND ` +
	`resource.type="bigquery_resource" AND NOT ` +
	`protoPayload.serviceData.jobCompletedEvent.job.jobConfiguration.query.query:(INFORMATION_SCHEMA OR __TABLES__) AND ` +
	`timestamp >= "%s" AND timestamp < "%s AND table_name"

This will return lesser logs for a specific table only. With this, we don't need to store the hash map and just do the collection on demand. The process would be

run extractor to extract table from bq > get a table > collect table stats for a specific time period and a specific table name > build table metadata > sink

We can add more filter to the query to only fetch the relevant logs.