diff --git a/docs/docs/concepts/recipe.md b/docs/docs/concepts/recipe.md index edb3a2634..3ce9d3f20 100644 --- a/docs/docs/concepts/recipe.md +++ b/docs/docs/concepts/recipe.md @@ -15,6 +15,7 @@ name: main-kafka-production # unique recipe name as an ID version: v1beta1 #recipe version source: # required - for fetching input from sources name: kafka # required - collector to use (e.g. bigquery, kafka) + scope: local-kafka # required - URN's namespace config: broker: "localhost:9092" sinks: # required - at least 1 sink defined @@ -34,13 +35,13 @@ processors: # optional - metadata processors Contains details about the ingredients of our recipe. The `config` of each source, sinks and processors differs as different data sources require different kinds of credentials, please refer more about them in further reference section. -| Key | Description | Requirement | further reference | -| :--- | :--- | :--- | :--- | -| `name` | **unique** recipe name, will be used as ID for job | required | N/A | -| `version` | Specify the version of recipe being used | required | N/A | -| `source` | contains details about the source of metadata extraction | required | [source](source.md) | -| `sinks` | defines the final destination of extracted and processed metadata | required | [sink](sink.md) | -| `processors` | used process the metadata before sinking | optional | [processor](processor.md) | +| Key | Description | Requirement | further reference | +|:-------------|:------------------------------------------------------------------|:------------|:--------------------------| +| `name` | **unique** recipe name, will be used as ID for job | required | N/A | +| `version` | Specify the version of recipe being used | required | N/A | +| `source` | contains details about the source of metadata extraction | required | [source](source.md) | +| `sinks` | defines the final destination of extracted and processed metadata | required | [sink](sink.md) | +| `processors` | used process the metadata before sinking | optional | [processor](processor.md) | ## Dynamic recipe value diff --git a/docs/docs/concepts/sink.md b/docs/docs/concepts/sink.md index 0f1fdb9b9..0be8a7d94 100644 --- a/docs/docs/concepts/sink.md +++ b/docs/docs/concepts/sink.md @@ -12,12 +12,9 @@ sinks: # required - at least 1 sink defined url: "https://example.com/metadata" - name: kafka config: - broker: localhost:9092 + brokers: localhost:9092 topic: "target-topic" - serializer: - type: proto - key: SampleLogKey - value: SampleLogMessage + key_path: ``` | key | Description | requirement | diff --git a/docs/docs/concepts/source.md b/docs/docs/concepts/source.md index 64e5fba04..cab6c92f9 100644 --- a/docs/docs/concepts/source.md +++ b/docs/docs/concepts/source.md @@ -1,6 +1,12 @@ # Source -When the source field is defined, Meteor will extract data from a metadata source using the details defined in the field. `type` field should define the name of Extractor you want, you can use one from this list [here](../reference/extractors.md). `config` of an extractor can be different for different Extractor and needs you to provide details to set up a connection between meteor and your source. To determine the required configurations you can visit README of each Extractor [here](https://github.com/odpf/meteor/tree/cb12c3ecf8904cf3f4ce365ca8981ccd132f35d0/plugins/extractors/README.md). +When the source field is defined, Meteor will extract data from a metadata +source using the details defined in the field. `name` field should define the +name of Extractor you want, you can use one from this list +[here](../reference/extractors.md). `config` of an extractor can be different +for different Extractor and needs you to provide details to set up a connection +between meteor and your source. To determine the required configurations you +can visit README of each Extractor [here](../reference/extractors.md). ## Writing source part of your recipe @@ -11,10 +17,10 @@ source: broker: broker:9092 ``` -| key | Description | requirement | -| :--- | :--- | :--- | -| `type` | contains the name of extractor, will be used for registry | required | -| `config` | different extractor will require different configuration | optional, depends on extractor | +| key | Description | requirement | +|:---------|:----------------------------------------------------------|:-------------------------------| +| `name` | contains the name of extractor, will be used for registry | required | +| `config` | different extractor will require different configuration | optional, depends on extractor | -To get more information about the list of extractors we have, and how to define `type` field refer [here](../reference/extractors.md). +To get more information about the list of extractors we have, and how to define `name` field refer [here](../reference/extractors.md). diff --git a/docs/docs/example/README.md b/docs/docs/example/README.md index 734ef1585..afe6041f6 100644 --- a/docs/docs/example/README.md +++ b/docs/docs/example/README.md @@ -7,8 +7,8 @@ export METEOR_KAFKA_BROKER=localhost:9092 meteor run ./kafka-console.yaml ``` -This recipe tells meteor to fetch kafka metadata from broker defined by `METEOR_KAFKA_BROKER` envvar which will be translated to `kafka_broker` in recipe. ([learn more about dynamic recipe](../docs/concepts/recipe.md#dynamic-recipe-value)) +This recipe tells meteor to fetch kafka metadata from broker defined by `METEOR_KAFKA_BROKER` envvar which will be translated to `kafka_broker` in recipe. ([learn more about dynamic recipe](../concepts/recipe.md#dynamic-recipe-value)) -[enrich](../docs/reference/processors.md#enrich) processor is also added in the `processors` list in the recipe which will enrich metadata fetched with keyvalues defined in the config. +[enrich](../reference/processors.md#enrich) processor is also added in the `processors` list in the recipe which will enrich metadata fetched with keyvalues defined in the config. -At last, Meteor will output the results to the `sinks` given in the recipe which in this case is to [console](../docs/reference/sinks.md#console) only. +At last, Meteor will output the results to the `sinks` given in the recipe which in this case is to [console](../reference/sinks.md#console) only. diff --git a/docs/docs/guides/0_installation.md b/docs/docs/guides/0_installation.md index 71dea46f1..efe13f53a 100644 --- a/docs/docs/guides/0_installation.md +++ b/docs/docs/guides/0_installation.md @@ -17,9 +17,9 @@ $ meteor list extractors ## Binary from releases -The binaries are downloadable at the releases tab. +The binaries are downloadable from the [Github releases][github-releases] page. There is currently no installer available. -You have to add the meteor binary to the PATH environment variable yourself or put the binary in a location that is already in your $PATH (e.g. /usr/local/bin, ...). +You have to add the meteor binary to the `PATH` environment variable yourself or put the binary in a location that is already in your `$PATH` (e.g. /usr/local/bin, ...). Once installed, you should be able to run: @@ -52,3 +52,5 @@ $ make build $ ./meteor --help ``` + +[github-releases]: https://github.com/odpf/meteor/releases diff --git a/docs/docs/reference/extractors.md b/docs/docs/reference/extractors.md index 7aee12c2d..af723045b 100644 --- a/docs/docs/reference/extractors.md +++ b/docs/docs/reference/extractors.md @@ -6,61 +6,88 @@ Meteor currently supports metadata extraction on these data sources. To perform ### Table -| Type | Attributes | Profile | Schema | Lineage | Ownership | Custom | -|:-----------------------------------------------------------------|:-----------|:--------|:-------|:--------|:----------|:-------| -| [`clickhouse`](../../../plugins/extractors/clickhouse/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`couchdb`](../../../plugins/extractors/couchdb/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`mongodb`](../../../plugins/extractors/mongodb/README.md) | ✅ | ✅ | ✗ | ✗ | ✗ | ✗ | -| [`mssql`](../../../plugins/extractors/mssql/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`mysql`](../../../plugins/extractors/mysql/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`postgres`](../../../plugins/extractors/postgres/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`cassandra`](../../../plugins/extractors/cassandra/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`oracle`](../../../plugins/extractors/oracle/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`mariadb`](../../../plugins/extractors/mariadb/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`redshift`](../../../plugins/extractors/redshift/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`presto`](../../../plugins/extractors/presto/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`snowflake`](../../../plugins/extractors/snowflake/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| Type | Attributes | Profile | Schema | Lineage | Ownership | Custom | +|:----------------------------------|:-----------|:--------|:-------|:--------|:----------|:-------| +| [`clickhouse`][clickhouse-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`couchdb`][couchdb-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`mongodb`][mongodb-readme] | ✅ | ✅ | ✗ | ✗ | ✗ | ✗ | +| [`mssql`][mssql-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`mysql`][mysql-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`postgres`][postgres-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`cassandra`][cassandra-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`oracle`][oracle-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`mariadb`][mariadb-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`redshift`][redshift-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`presto`][presto-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`snowflake`][snowflake-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ | ### Dashboard -| Type | Url | Chart | Lineage | Tags | Custom | -|:-------------------------------------------------------------|:----|:------|:--------|:-----|:-------| -| [`grafana`](../../../plugins/extractors/grafana/README.md) | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`metabase`](../../../plugins/extractors/metabase/README.md) | ✅ | ✅ | ✗ | ✗ | ✗ | -| [`superset`](../../../plugins/extractors/superset/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | -| [`tableau`](../../../plugins/extractors/tableau/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | -| [`redash`](../../../plugins/extractors/redash/README.md) | ✅ | ✗ | ✗ | ✗ | ✗ | +| Type | Url | Chart | Lineage | Tags | Custom | +|:------------------------------|:----|:------|:--------|:-----|:-------| +| [`grafana`][grafana-readme] | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`metabase`][metabase-readme] | ✅ | ✅ | ✗ | ✗ | ✗ | +| [`superset`][superset-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | +| [`tableau`][tableau-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | +| [`redash`][redash-readme] | ✅ | ✗ | ✗ | ✗ | ✗ | ### Topic -| Type | Profile | Schema | Ownership | Lineage | Tags | Custom | -|:-------------------------------------------------------|:--------|:-------|:----------|:--------|:-----|:-------| -| [`kafka`](../../../plugins/extractors/kafka/README.md) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| Type | Profile | Schema | Ownership | Lineage | Tags | Custom | +|:------------------------|:--------|:-------|:----------|:--------|:-----|:-------| +| [`kafka`][kafka-readme] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ### User -| Type | Email | Username | FullName | Title | IsActive | ManagerEmail | Profiles | Memberships | facets | common | -|:---------------------------------------------------------|:------|:---------|:---------|:------|:---------|:-------------|:---------|:------------|:-------|:-------| -| [`github`](../../../plugins/extractors/github/README.md) | ✅ | ✅ | ✅ | ☐ | ✅ | ☐ | ☐ | ☐ | ☐ | ☐ | -| [`shield`](../../../plugins/extractors/shield/README.md) | ✅ | ✅ | ✅ | ☐ | ✅ | ☐ | ☐ | ✅ | ✅ | ☐ | -| [`gsuite`](../../../plugins/extractors/gsuite/README.md) | ✅ | ☐ | ✅ | ☐ | ✅ | ✅ | ☐ | ☐ | ☐ | ☐ | +| Type | Email | Username | FullName | Title | IsActive | ManagerEmail | Profiles | Memberships | facets | common | +|:--------------------------|:------|:---------|:---------|:------|:---------|:-------------|:---------|:------------|:-------|:-------| +| [`github`][github-readme] | ✅ | ✅ | ✅ | ✗ | ✅ | ✗ | ✗ | ✗ | ✗ | ✗ | +| [`shield`][shield-readme] | ✅ | ✅ | ✅ | ✗ | ✅ | ✗ | ✗ | ✅ | ✅ | ✗ | +| [`gsuite`][gsuite-readme] | ✅ | ✗ | ✅ | ✗ | ✅ | ✅ | ✗ | ✗ | ✗ | ✗ | ### Bucket -| type | Location | StorageType | Blobs | Ownership | Tags | Custom | Timestamps | -|:---------------------------------------------------|:---------|:------------|:------|:----------|:-----|:-------|:-----------| -| [`gcs`](../../../plugins/extractors/gcs/README.md) | ✅ | ✅ | ✗ | ✅ | ✅ | ✗ | ✅ | +| Type | Location | StorageType | Blobs | Ownership | Tags | Custom | Timestamps | +|:--------------------|:---------|:------------|:------|:----------|:-----|:-------|:-----------| +| [`gcs`][gcs-readme] | ✅ | ✅ | ✗ | ✅ | ✅ | ✗ | ✅ | ### Job -| Type | Ownership | Upstreams | Downstreams | Custom | -|:-----------------------------------------------------------|:----------|:----------|:------------|:-------| -| [`optimus`](../../../plugins/extractors/optimus/README.md) | ✅ | ✅ | ✅ | ✅ | ✅ | +| Type | Ownership | Upstreams | Downstreams | Custom | +|:----------------------------|:----------|:----------|:------------|:-------| +| [`optimus`][optimus-readme] | ✅ | ✅ | ✅ | ✅ | ✅ | ### Machine Learning Feature Table -| Type | Ownership | Upstreams | Downstreams | Custom | -|:-------------------------------------------------------------------|:----------|:----------|:------------|:-------| -| [`caramlstore`](../../../plugins/extractors/caramlstore/README.md) | ✗ | ✅ | ✗ | ✅ | - +| Type | Ownership | Upstreams | Downstreams | Custom | +|:------------------------------------|:----------|:----------|:------------|:-------| +| [`caramlstore`][caramlstore-readme] | ✗ | ✅ | ✗ | ✅ | + + + + +[clickhouse-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/clickhouse/README.md +[couchdb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/couchdb/README.md +[mongodb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mongodb/README.md +[mssql-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mssql/README.md +[mysql-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mysql/README.md +[postgres-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/postgres/README.md +[cassandra-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/cassandra/README.md +[oracle-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/oracle/README.md +[mariadb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mariadb/README.md +[redshift-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/redshift/README.md +[presto-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/presto/README.md +[snowflake-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/snowflake/README.md +[grafana-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/grafana/README.md +[metabase-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/metabase/README.md +[superset-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/superset/README.md +[tableau-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/tableau/README.md +[redash-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/redash/README.md +[kafka-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/kafka/README.md +[github-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/github/README.md +[shield-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/shield/README.md +[gsuite-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/gsuite/README.md +[gcs-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/gcs/README.md +[optimus-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/optimus/README.md +[caramlstore-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/caramlstore/README.md