Skip to content

fix: don't use relative paths for links outside docs fldr #422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ravisuhag merged 1 commit into from
Oct 13, 2022
Merged

fix: don't use relative paths for links outside docs fldr #422

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions docs/docs/concepts/recipe.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ name: main-kafka-production # unique recipe name as an ID
version: v1beta1 #recipe version
source: # required - for fetching input from sources
name: kafka # required - collector to use (e.g. bigquery, kafka)
scope: local-kafka # required - URN's namespace
config:
broker: "localhost:9092"
sinks: # required - at least 1 sink defined
Expand All @@ -34,13 +35,13 @@ processors: # optional - metadata processors

Contains details about the ingredients of our recipe. The `config` of each source, sinks and processors differs as different data sources require different kinds of credentials, please refer more about them in further reference section.

| Key | Description | Requirement | further reference |
| :--- | :--- | :--- | :--- |
| `name` | **unique** recipe name, will be used as ID for job | required | N/A |
| `version` | Specify the version of recipe being used | required | N/A |
| `source` | contains details about the source of metadata extraction | required | [source](source.md) |
| `sinks` | defines the final destination of extracted and processed metadata | required | [sink](sink.md) |
| `processors` | used process the metadata before sinking | optional | [processor](processor.md) |
| Key | Description | Requirement | further reference |
|:-------------|:------------------------------------------------------------------|:------------|:--------------------------|
| `name` | **unique** recipe name, will be used as ID for job | required | N/A |
| `version` | Specify the version of recipe being used | required | N/A |
| `source` | contains details about the source of metadata extraction | required | [source](source.md) |
| `sinks` | defines the final destination of extracted and processed metadata | required | [sink](sink.md) |
| `processors` | used process the metadata before sinking | optional | [processor](processor.md) |

## Dynamic recipe value

Expand Down
7 changes: 2 additions & 5 deletions docs/docs/concepts/sink.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,9 @@ sinks: # required - at least 1 sink defined
url: "https://example.com/metadata"
- name: kafka
config:
broker: localhost:9092
brokers: localhost:9092
topic: "target-topic"
serializer:
type: proto
key: SampleLogKey
value: SampleLogMessage
key_path:
```

| key | Description | requirement |
Expand Down
18 changes: 12 additions & 6 deletions docs/docs/concepts/source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
# Source

When the source field is defined, Meteor will extract data from a metadata source using the details defined in the field. `type` field should define the name of Extractor you want, you can use one from this list [here](../reference/extractors.md). `config` of an extractor can be different for different Extractor and needs you to provide details to set up a connection between meteor and your source. To determine the required configurations you can visit README of each Extractor [here](https://github.com/odpf/meteor/tree/cb12c3ecf8904cf3f4ce365ca8981ccd132f35d0/plugins/extractors/README.md).
When the source field is defined, Meteor will extract data from a metadata
source using the details defined in the field. `name` field should define the
name of Extractor you want, you can use one from this list
[here](../reference/extractors.md). `config` of an extractor can be different
for different Extractor and needs you to provide details to set up a connection
between meteor and your source. To determine the required configurations you
can visit README of each Extractor [here](../reference/extractors.md).

## Writing source part of your recipe

Expand All @@ -11,10 +17,10 @@ source:
broker: broker:9092
```

| key | Description | requirement |
| :--- | :--- | :--- |
| `type` | contains the name of extractor, will be used for registry | required |
| `config` | different extractor will require different configuration | optional, depends on extractor |
| key | Description | requirement |
|:---------|:----------------------------------------------------------|:-------------------------------|
| `name` | contains the name of extractor, will be used for registry | required |
| `config` | different extractor will require different configuration | optional, depends on extractor |

To get more information about the list of extractors we have, and how to define `type` field refer [here](../reference/extractors.md).
To get more information about the list of extractors we have, and how to define `name` field refer [here](../reference/extractors.md).

6 changes: 3 additions & 3 deletions docs/docs/example/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ export METEOR_KAFKA_BROKER=localhost:9092
meteor run ./kafka-console.yaml
```

This recipe tells meteor to fetch kafka metadata from broker defined by `METEOR_KAFKA_BROKER` envvar which will be translated to `kafka_broker` in recipe. ([learn more about dynamic recipe](../docs/concepts/recipe.md#dynamic-recipe-value))
This recipe tells meteor to fetch kafka metadata from broker defined by `METEOR_KAFKA_BROKER` envvar which will be translated to `kafka_broker` in recipe. ([learn more about dynamic recipe](../concepts/recipe.md#dynamic-recipe-value))

[enrich](../docs/reference/processors.md#enrich) processor is also added in the `processors` list in the recipe which will enrich metadata fetched with keyvalues defined in the config.
[enrich](../reference/processors.md#enrich) processor is also added in the `processors` list in the recipe which will enrich metadata fetched with keyvalues defined in the config.

At last, Meteor will output the results to the `sinks` given in the recipe which in this case is to [console](../docs/reference/sinks.md#console) only.
At last, Meteor will output the results to the `sinks` given in the recipe which in this case is to [console](../reference/sinks.md#console) only.
6 changes: 4 additions & 2 deletions docs/docs/guides/0_installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ $ meteor list extractors

## Binary from releases

The binaries are downloadable at the releases tab.
The binaries are downloadable from the [Github releases][github-releases] page.
There is currently no installer available.
You have to add the meteor binary to the PATH environment variable yourself or put the binary in a location that is already in your $PATH (e.g. /usr/local/bin, ...).
You have to add the meteor binary to the `PATH` environment variable yourself or put the binary in a location that is already in your `$PATH` (e.g. /usr/local/bin, ...).

Once installed, you should be able to run:

Expand Down Expand Up @@ -52,3 +52,5 @@ $ make build

$ ./meteor --help
```

[github-releases]: https://github.com/odpf/meteor/releases
105 changes: 66 additions & 39 deletions docs/docs/reference/extractors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,61 +6,88 @@ Meteor currently supports metadata extraction on these data sources. To perform

### Table

| Type | Attributes | Profile | Schema | Lineage | Ownership | Custom |
|:-----------------------------------------------------------------|:-----------|:--------|:-------|:--------|:----------|:-------|
| [`clickhouse`](../../../plugins/extractors/clickhouse/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`couchdb`](../../../plugins/extractors/couchdb/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`mongodb`](../../../plugins/extractors/mongodb/README.md) | ✅ | ✅ | ✗ | ✗ | ✗ | ✗ |
| [`mssql`](../../../plugins/extractors/mssql/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`mysql`](../../../plugins/extractors/mysql/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`postgres`](../../../plugins/extractors/postgres/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`cassandra`](../../../plugins/extractors/cassandra/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`oracle`](../../../plugins/extractors/oracle/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`mariadb`](../../../plugins/extractors/mariadb/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`redshift`](../../../plugins/extractors/redshift/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`presto`](../../../plugins/extractors/presto/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`snowflake`](../../../plugins/extractors/snowflake/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| Type | Attributes | Profile | Schema | Lineage | Ownership | Custom |
|:----------------------------------|:-----------|:--------|:-------|:--------|:----------|:-------|
| [`clickhouse`][clickhouse-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`couchdb`][couchdb-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`mongodb`][mongodb-readme] | ✅ | ✅ | ✗ | ✗ | ✗ | ✗ |
| [`mssql`][mssql-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`mysql`][mysql-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`postgres`][postgres-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`cassandra`][cassandra-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`oracle`][oracle-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`mariadb`][mariadb-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`redshift`][redshift-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`presto`][presto-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`snowflake`][snowflake-readme] | ✅ | ✅ | ✅ | ✗ | ✗ | ✗ |

### Dashboard

| Type | Url | Chart | Lineage | Tags | Custom |
|:-------------------------------------------------------------|:----|:------|:--------|:-----|:-------|
| [`grafana`](../../../plugins/extractors/grafana/README.md) | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`metabase`](../../../plugins/extractors/metabase/README.md) | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`superset`](../../../plugins/extractors/superset/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ |
| [`tableau`](../../../plugins/extractors/tableau/README.md) | ✅ | ✅ | ✅ | ✗ | ✗ |
| [`redash`](../../../plugins/extractors/redash/README.md) | ✅ | ✗ | ✗ | ✗ | ✗ |
| Type | Url | Chart | Lineage | Tags | Custom |
|:------------------------------|:----|:------|:--------|:-----|:-------|
| [`grafana`][grafana-readme] | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`metabase`][metabase-readme] | ✅ | ✅ | ✗ | ✗ | ✗ |
| [`superset`][superset-readme] | ✅ | ✅ | ✅ | ✗ | ✗ |
| [`tableau`][tableau-readme] | ✅ | ✅ | ✅ | ✗ | ✗ |
| [`redash`][redash-readme] | ✅ | ✗ | ✗ | ✗ | ✗ |


### Topic

| Type | Profile | Schema | Ownership | Lineage | Tags | Custom |
|:-------------------------------------------------------|:--------|:-------|:----------|:--------|:-----|:-------|
| [`kafka`](../../../plugins/extractors/kafka/README.md) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Type | Profile | Schema | Ownership | Lineage | Tags | Custom |
|:------------------------|:--------|:-------|:----------|:--------|:-----|:-------|
| [`kafka`][kafka-readme] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |

### User

| Type | Email | Username | FullName | Title | IsActive | ManagerEmail | Profiles | Memberships | facets | common |
|:---------------------------------------------------------|:------|:---------|:---------|:------|:---------|:-------------|:---------|:------------|:-------|:-------|
| [`github`](../../../plugins/extractors/github/README.md) | ✅ | ✅ | ✅ | ☐ | ✅ | ☐ | ☐ | ☐ | ☐ | ☐ |
| [`shield`](../../../plugins/extractors/shield/README.md) | ✅ | ✅ | ✅ | ☐ | ✅ | ☐ | ☐ | ✅ | ✅ | ☐ |
| [`gsuite`](../../../plugins/extractors/gsuite/README.md) | ✅ | ☐ | ✅ | ☐ | ✅ | ✅ | ☐ | ☐ | ☐ | ☐ |
| Type | Email | Username | FullName | Title | IsActive | ManagerEmail | Profiles | Memberships | facets | common |
|:--------------------------|:------|:---------|:---------|:------|:---------|:-------------|:---------|:------------|:-------|:-------|
| [`github`][github-readme] | ✅ | ✅ | ✅ | ✗ | ✅ | ✗ | ✗ | ✗ | ✗ | ✗ |
| [`shield`][shield-readme] | ✅ | ✅ | ✅ | ✗ | ✅ | ✗ | ✗ | ✅ | ✅ | ✗ |
| [`gsuite`][gsuite-readme] | ✅ | ✗ | ✅ | ✗ | ✅ | ✅ | ✗ | ✗ | ✗ | ✗ |

### Bucket

| type | Location | StorageType | Blobs | Ownership | Tags | Custom | Timestamps |
|:---------------------------------------------------|:---------|:------------|:------|:----------|:-----|:-------|:-----------|
| [`gcs`](../../../plugins/extractors/gcs/README.md) | ✅ | ✅ | ✗ | ✅ | ✅ | ✗ | ✅ |
| Type | Location | StorageType | Blobs | Ownership | Tags | Custom | Timestamps |
|:--------------------|:---------|:------------|:------|:----------|:-----|:-------|:-----------|
| [`gcs`][gcs-readme] | ✅ | ✅ | ✗ | ✅ | ✅ | ✗ | ✅ |

### Job

| Type | Ownership | Upstreams | Downstreams | Custom |
|:-----------------------------------------------------------|:----------|:----------|:------------|:-------|
| [`optimus`](../../../plugins/extractors/optimus/README.md) | ✅ | ✅ | ✅ | ✅ | ✅ |
| Type | Ownership | Upstreams | Downstreams | Custom |
|:----------------------------|:----------|:----------|:------------|:-------|
| [`optimus`][optimus-readme] | ✅ | ✅ | ✅ | ✅ | ✅ |

### Machine Learning Feature Table

| Type | Ownership | Upstreams | Downstreams | Custom |
|:-------------------------------------------------------------------|:----------|:----------|:------------|:-------|
| [`caramlstore`](../../../plugins/extractors/caramlstore/README.md) | ✗ | ✅ | ✗ | ✅ |

| Type | Ownership | Upstreams | Downstreams | Custom |
|:------------------------------------|:----------|:----------|:------------|:-------|
| [`caramlstore`][caramlstore-readme] | ✗ | ✅ | ✗ | ✅ |


<!--- Not using relative links because that breaks the docs build -->

[clickhouse-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/clickhouse/README.md
[couchdb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/couchdb/README.md
[mongodb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mongodb/README.md
[mssql-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mssql/README.md
[mysql-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mysql/README.md
[postgres-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/postgres/README.md
[cassandra-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/cassandra/README.md
[oracle-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/oracle/README.md
[mariadb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mariadb/README.md
[redshift-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/redshift/README.md
[presto-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/presto/README.md
[snowflake-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/snowflake/README.md
[grafana-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/grafana/README.md
[metabase-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/metabase/README.md
[superset-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/superset/README.md
[tableau-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/tableau/README.md
[redash-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/redash/README.md
[kafka-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/kafka/README.md
[github-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/github/README.md
[shield-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/shield/README.md
[gsuite-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/gsuite/README.md
[gcs-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/gcs/README.md
[optimus-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/optimus/README.md
[caramlstore-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/caramlstore/README.md