Skip to content

Commit 1226406

Browse files
sudo-suhassudo-suhas
committed
docs: fix minor issues
- application_yaml extractor: - Fix usage for - format for 'input' and 'output' fields. - Drop fields lineage.{upstreams, downstreams}[].{type, service} from outputs section. - In the bigquery extractor, replace reference to 'credentials_json' with 'service_account_json' and 'service_account_base64'. - In the caramlstore extractor, fix the value format and sample for 'resource.urn' in outputs section.
1 parent d3cde4e commit 1226406

3 files changed

Lines changed: 22 additions & 33 deletions

File tree

‎plugins/extractors/application_yaml/README.md‎

Lines changed: 7 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -33,21 +33,12 @@ description: "string"
3333
url: "string"
3434
version: "string"
3535
inputs: # OPTIONAL
36-
- type: "batch" # ONE OF {batch, kafka}
37-
platform: "bigquery" # If type is batch then bigquery
38-
options:
39-
table_ref: "{project_id}:{dataset_id}.{table_id}" # REQUIRED
40-
- type: "stream"
41-
platform: "kafka" # If type is stream then kafka
42-
options:
43-
topic: "estimate-log" # REQUIRED
44-
bootstrap_servers: "host:port" # REQUIRED
36+
# Format: "urn:{service}:{scope}:{type}:{name}"
37+
- urn:bigquery:bq-raw-internal:table:bq-raw-internal:dagstream.production_feast09_s2id13_30min_demand
38+
- urn:kafka:int-dagstream-kafka.yonkou.io:topic:staging_feast09_s2id13_30min_demand
4539
outputs: # OPTIONAL
46-
- type: "stream"
47-
platform: "kafka"
48-
options:
49-
topic: "string"
50-
bootstrap_servers: "host:port"
40+
# Format: "urn:{service}:{scope}:{type}:{name}"
41+
- urn:kafka:1-my-kafka.com:topic:staging_feast09_mixed_granularity_demand_forecast_3es
5142
create_time: "2006-01-02T15:04:05Z"
5243
update_time: "2006-01-02T15:04:05Z"
5344
labels:
@@ -90,12 +81,8 @@ proto definitions for more information.
9081
| `ownership.owners[0].urn` | `{application.team.id}` | `9ebcc2f8-5894-47c6-83a9-160b7eaa3f6b` |
9182
| `ownership.owners[0].name` | `{application.team.name}` | `Search` |
9283
| `ownership.owners[0].email` | `{application.team.email}` | `search@mycompany.com` |
93-
| `lineage.upstreams[].urn` | `urn:{service}:{scope}:{type}:{name}` | `urn:kafka:int-kafka.yonkou.io:topic:staging_30min_demand` |
94-
| `lineage.upstreams[].type` | `{type}` | `topic` |
95-
| `lineage.upstreams[].service` | `{service}` | `kafka` |
96-
| `lineage.downstreams[].urn` | `urn:{service}:{scope}:{type}:{name}` | `urn:bigquery:bq-internal:table:bq-internal:dagstream.production_30min_demand` |
97-
| `lineage.downstreams[].type` | `{type}` | `table` |
98-
| `lineage.downstreams[].service` | `{service}` | `bigquery` |
84+
| `lineage.upstreams[].urn` | `{application.inputs[]}` | `urn:kafka:int-kafka.yonkou.io:topic:staging_30min_demand` |
85+
| `lineage.downstreams[].urn` | `{application.outputs[]}` | `urn:bigquery:bq-internal:table:bq-internal:dagstream.production_30min_demand` |
9986
| `resource.labels` | `map[string]string` | `{"team": "Booking Experience"}` |
10087

10188
## Contributing

‎plugins/extractors/bigquery/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,8 @@ source:
4747

4848
### *Notes*
4949

50-
- Leaving `credentials_json` blank will default
51-
to [Google's default authentication](https://cloud.google.com/docs/authentication/production#automatically). It is
50+
- Leaving `service_account_json` and `service_account_base64` blank will default
51+
to [Google's default authentication][google-default-auth]. It is
5252
recommended if Meteor instance runs inside the same Google Cloud environment as the BigQuery project.
5353
- Service account needs to have `bigquery.privateLogsViewer` role to be able to collect bigquery audit logs
5454

@@ -89,3 +89,5 @@ source:
8989

9090
Refer to the [contribution guidelines](../../../docs/docs/contribute/guide.md#adding-a-new-extractor) for information on
9191
contributing to this module.
92+
93+
[google-default-auth]: https://cloud.google.com/docs/authentication/production#automatically

‎plugins/extractors/caramlstore/README.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -31,17 +31,17 @@ The feature tables are mapped to an [`Asset`][proton-asset] with model specific
3131
metadata stored using [`FeatureTable`][proton-featuretable]. Please refer
3232
the proto definitions for more information.
3333

34-
| Field | Value | Sample Value |
35-
|:-------------------|:----------------------------------------------------------------|:---------------------------------------------------------------------------|
36-
| `resource.urn` | `urn:caramlstore:{scope}:ml-feature-table:{feature_table_name}` | `urn:caramlstore:caramlstore-stg:feature_table:merchant_uuid_t2_discovery` |
37-
| `resource.name` | `{feature_table.name}` | `merchant_uuid_t2_discovery` |
38-
| `resource.service` | `caramlstore` | `caramlstore` |
39-
| `resource.type` | `feature_table` | `feature_table` |
40-
| `namespace` | `{feature_table.project}` | `my_project` |
41-
| `entities` | [`[]Entity`](#entity) | |
42-
| `features` | [`[]Feature`](#feature) | |
43-
| `create_time` | `{feature_table.created_timestamp}` | `2022-08-08T03:17:54Z` |
44-
| `update_time` | `{feature_table.updated_timestamp}` | `2022-08-08T03:57:54Z` |
34+
| Field | Value | Sample Value |
35+
|:-------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
36+
| `resource.urn` | `urn:caramlstore:{scope}:feature_table:{feature_table.project}.{feature_table.name}` | `urn:caramlstore:caramlstore-stg:feature_table:my_project.merchant_uuid_t2_discovery` |
37+
| `resource.name` | `{feature_table.name}` | `merchant_uuid_t2_discovery` |
38+
| `resource.service` | `caramlstore` | `caramlstore` |
39+
| `resource.type` | `feature_table` | `feature_table` |
40+
| `namespace` | `{feature_table.project}` | `my_project` |
41+
| `entities` | [`[]Entity`](#entity) | |
42+
| `features` | [`[]Feature`](#feature) | |
43+
| `create_time` | `{feature_table.created_timestamp}` | `2022-08-08T03:17:54Z` |
44+
| `update_time` | `{feature_table.updated_timestamp}` | `2022-08-08T03:57:54Z` |
4545

4646
### Entity
4747

0 commit comments

Comments
 (0)