Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/concepts/processor.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Processor

A recipe can have none or many processors registered, depending upon the way user want metadata to processed. A processor is basically a function that:
A recipe can have none or many processors registered, depending upon the way the user wants metadata to be processed. A processor is basically a function that:

* expects a list of data
* processes the list
Expand Down
16 changes: 8 additions & 8 deletions docs/docs/concepts/recipe.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Recipe

A recipe is a set of instructions and configurations defined by user, and in Meteor they are used to define how a particular job will be performed. It should contain instruction about the `source` from which the metadata will be fetched, information about metadata `processors` and the destination is to be defined as `sinks` of metadata.
A recipe is a set of instructions and configurations defined by the user, and in Meteor they are used to define how a particular job will be performed. It should contain instructions about the `source` from which the metadata will be fetched, information about metadata `processors` and the destination is to be defined as `sinks` of metadata.

The recipe should contain about **only one** `source` since we wish to have a seperate job for different extractors, hence keeping them isolated. Should have **atleast one** destination of metadata mentioned in `sinks`, and the `processors` field is optional but can have multiple processors.
The recipe should contain about **only one** `source` since we wish to have a seperate job for different extractors, hence keeping them isolated. Should have **at least one** destination of metadata mentioned in `sinks`, and the `processors` field is optional but can have multiple processors.

Recipe is a yaml file, follows a structure as shown below and needs to passed as a individual file or as a bunch of recipes contained in a directory as shown in [sample usage](recipe.md#sample-usage).
Recipe is a yaml file, follows a structure as shown below and needs to be passed as an individual file or as a bunch of recipes contained in a directory as shown in [sample usage](recipe.md#sample-usage).

## Writing a Recipe for Meteor

Expand Down Expand Up @@ -32,19 +32,19 @@ processors: # optional - metadata processors

### Glossary Table

Contains details about the ingridients of our recipe. The `config` of each source, sinks and processors differs as different data source required different kinds of credentials, please refer more about them in further reference section.
Contains details about the ingredients of our recipe. The `config` of each source, sinks and processors differs as different data sources require different kinds of credentials, please refer more about them in further reference section.

| Key | Description | Requirement | further reference |
| :--- | :--- | :--- | :--- |
| `name` | **unique** recipe name, will be used as ID for job | required | N/A |
| `version` | Specify the version of recipe being used | required | N/A |
| `source` | contains details about the source of metadata extraction | required | [source](source.md) |
| `sinks` | defines the final destination's of extracted and processed metadata | required | [sink](sink.md) |
| `sinks` | defines the final destination of extracted and processed metadata | required | [sink](sink.md) |
| `processors` | used process the metadata before sinking | optional | [processor](processor.md) |

## Dynamic recipe value

Meteor reads recipe using [go template](https://golang.org/pkg/text/template/), which means you can put a variable instead of static value in a recipe.
Meteor reads recipe using [go template](https://golang.org/pkg/text/template/), which means you can put a variable instead of a static value in a recipe.
Environment variables with prefix `METEOR_`, such as `METEOR_MONGODB_PASS`, will be used as the template data for the recipe.
This is to allow you to skip creating recipes containing the credentials of datasource.

Expand Down Expand Up @@ -78,9 +78,9 @@ sinks:

## Support to pass env variables with --var flag

Meteor allows you to maintain a `.yaml` file as well which can be used as a template data for recipe.
Meteor allows you to maintain a `.yaml` file as well which can be used as template data for recipes.
The variables here should not contain a `METEOR_` prefix and should be as normal as any other `config` file.
Meteor reads both local environment variables as well as the ones from `yaml` file and in case of conflict prefers the one mentioned in `yaml` file.
Meteor reads both local environment variables as well as the ones from the `yaml` file and in case of conflict prefers the one mentioned in the `yaml` file.

* _sample-config.yaml_

Expand Down
6 changes: 3 additions & 3 deletions docs/docs/concepts/sink.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Sink

`sinks` are used to define the medium of consuming the metadata being extracted. You need to specify **atleast one** sink or can specify multiple sinks in a recipe, this will prevent you from having to create duplicate recipes for the same job. The given examples shows you its correct usage if your sink is `http` and `kafka`.
`sinks` are used to define the medium of consuming the metadata being extracted. You need to specify **at least one** sink or can specify multiple sinks in a recipe, this will prevent you from having to create duplicate recipes for the same job. The given examples show you its correct usage if your sink is `http` and `kafka`.

## Writing `sinks` part of your recipe

Expand Down Expand Up @@ -60,9 +60,9 @@ By default, metadata would be serialized into JSON format before sinking. To sen

## Custom Sink

Meteor has built-in sinks like Kafka and HTTP which users can just utilise directly. We will also allow creating custom sinks for DRY purposes.
Meteor has built-in sinks like Kafka and HTTP which users can just use directly. We will also allow creating custom sinks for DRY purposes.

It will be useful if you can find yourself sinking multiple metadata source to one place.
It will be useful if you can find yourself sinking multiple metadata sources to one place.

### Sample Custom Sink

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/concepts/source.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Source

When the source field is defined, Meteor will extract data from a metadata source using the details defined in the field. `type` field should define the name of Extractor you want, you can use one from this list [here](../reference/extractors.md). `config` of a extractor can be different for different Extractor and needs you to provide details to setup a connection between meteor and your source. To determine the required configurations you can visit README of each Extractor [here](https://github.com/odpf/meteor/tree/cb12c3ecf8904cf3f4ce365ca8981ccd132f35d0/plugins/extractors/README.md).
When the source field is defined, Meteor will extract data from a metadata source using the details defined in the field. `type` field should define the name of Extractor you want, you can use one from this list [here](../reference/extractors.md). `config` of an extractor can be different for different Extractor and needs you to provide details to set up a connection between meteor and your source. To determine the required configurations you can visit README of each Extractor [here](https://github.com/odpf/meteor/tree/cb12c3ecf8904cf3f4ce365ca8981ccd132f35d0/plugins/extractors/README.md).

## Writing source part of your recipe

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/guides/1_list_Plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Before getting started we expect you went through the [prerequisites](./introduc
Meteor follows a plugin driven approach and hence includes basically three kinds of plugins for the metadata orchestration: extractors (source), processors, and sinks (destination).
Some details on these 3 are:

- **Extractors** are the set of plugins that are source of our metadata and include databases, dashboards, users, etc.
- **Extractors** are the set of plugins that are the source of our metadata and include databases, dashboards, users, etc.

- **Processors** are the set of plugins that perform the enrichment or data processing for the metadata after extraction.

Expand Down
6 changes: 3 additions & 3 deletions docs/docs/guides/2_manage_recipes.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Recipes - Creation and linting

A recipe is a set of instructions and configurations defined by user, and in Meteor they are used to define how a particular job will be performed.
A recipe is a set of instructions and configurations defined by the user, and in Meteor they are used to define how a particular job will be performed.
Thus, for the entire set of orchestration all you will need to provide will be recipe\(s\) for all the jobs you want meteor to do.

Read more about the concepts of Recipe [here](../concepts/recipe.md).
Expand All @@ -9,7 +9,7 @@ A sample recipe can be generated using the `meteor new` command mentioned [below

One can also generate multiple recipes with similar configurations using the `meteor gen` command mentioned [below](#generating-multiple-recipes-from-a-template).

After making the necessary changes to the source, and sinks as per your local setup, you can validate the sample-recipe using steps mentioed [here](#linting-recipes).
After making the necessary changes to the source, and sinks as per your local setup, you can validate the sample-recipe using steps mentioned [here](#linting-recipes).

## Generating new Sample recipe\(s\)

Expand All @@ -22,7 +22,7 @@ $ meteor new recipe sample -e <name-of-extractor> -s <single-or-multiple-sinks>
$ meteor new recipe sample -e bigquery -s compass,kafka

# for the tour you can use a single console sink
# extracor(-e) as postgres, sink(-s) and enrich processor(-p)
# extractor(-e) as postgres, sink(-s) and enrich processor(-p)
# save the generated recipe to a recipe.yaml
$ meteor new recipe sample -e postgres -s console -p enrich > recipe.yaml

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/guides/4_deployment.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Deployment

After we are done with running and veryfing that the recipes works with the data-source and sink you have mentioned.
After we are done with running and verifying that the recipes works with the data-source and sink you have mentioned.
You may want to automate the process of metadata collection on some regular basis as a cron job.
One can setup user for the same.

Expand Down
4 changes: 2 additions & 2 deletions docs/docs/guides/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ The tour takes you through how you can use meteor as an end user and is a great

- Install Homebrew (for macOS) or Docker (if using docker image).

- You are required to have atleast one Database, Dashboard or any other Data source and their proper credentials ready.
- You are required to have at least one Database, Dashboard or any other Data source and their proper credentials ready.
For e.g. Postgres, MySQL, MSSQL, MongoDB, GitHub, etc.

- For sink we will be using console, if you are planning to use compass or Kafka please make necessary changes accordingly.

Once done with the pre-requisites Get started with [Installation](./0_installation.md).
Once done with the prerequisites Get started with [Installation](./0_installation.md).
12 changes: 6 additions & 6 deletions docs/docs/reference/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@

Meteor currently supports the following commands and these can be utilised after the installation:

* completion: generate the autocompletion script for the specified shell
* completion: generate the auto completion script for the specified shell

* [gen](#creating-sample-recipes): The recipe will be printed on standard output.
Specify recipe name with the first argument without extension.
Use commma to separate multiple sinks and processors.
Use comma to separate multiple sinks and processors.

* [help](#get-help-on-commands-when-stuck): to help user with meteor.
* [help](#get-help-on-commands-when-stuck): to help the user with meteor.

* [info](#getting-information-about-plugins): Info command is used to get suitable information about various plugins.
Specify the type of plugin as extractor, sink or processor.
Expand All @@ -19,7 +19,7 @@ Helps in avoiding any failure during running the meteor due to invalid recipe fo

* [list](#listing-all-the-plugins): used to state all the plugins of a certain type.

* [run](#running-recipes): the command is used for running the metadata extraction as per the instructions in recipe.
* [run](#running-recipes): the command is used for running the metadata extraction as per the instructions in the recipe.
Can be used to run a single recipe, a directory of recipes or all the recipes in the current directory.

## Listing all the plugins
Expand Down Expand Up @@ -59,7 +59,7 @@ $ meteor info extractor postgres
## Generating Sample recipe\(s\)

Since recipe is the main resource of Meteor, we first need to create it before anything else.
You can create a sample recipe usin the gen command.
You can create a sample recipe using the gen command.

```bash
# generate a sample recipe
Expand All @@ -69,7 +69,7 @@ $ meteor gen recipe sample -e bigquery -s console
# generate recipe with multiple sinks
$ meteor gen recipe sample -e bigquery -s compass,kafka

# extracor(-e) as postgres, multiple sinks(-s) and enrich processor(-p)
# extractor(-e) as postgres, multiple sinks(-s) and enrich processor(-p)
# save the generated recipe to a recipe.yaml
meteor gen recipe sample -e postgres -s compass,kafka -p enrich > recipe.yaml
```
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/reference/configuration.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Configuration

This page contains reference for all the application configurations for Meteor.
This page contains references for all the application configurations for Meteor.

## Table of Contents

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/reference/extractors.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Extractors

Meteor currently support metadata extraction on these data sources. To perform extraction on any of these you need to create a recipe file with instructions as mentioned [here](../concepts/recipe.md). In the `sample-recipe.yaml` add `source` information such as `type` from the table below and `config` for that particular extractor can be found by visiting the link in `type` field.
Meteor currently supports metadata extraction on these data sources. To perform extraction on any of these you need to create a recipe file with instructions as mentioned [here](../concepts/recipe.md). In the `sample-recipe.yaml` add `source` information such as `type` from the table below and `config` for that particular extractor can be found by visiting the link in the `type` field.

## Extractors Feature Matrix

Expand Down
14 changes: 7 additions & 7 deletions docs/docs/reference/metadata_models.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,27 +9,27 @@ We are currently using the following metadata models:
Currently, Meteor provides a metadata extractor for the buckets mentioned [here](../reference/extractors.md)

- [Dashboard](https://github.com/odpf/proton/blob/main/odpf/assets/dashboard.proto):
Dashboards are essential part of data analysis and are used to track, analyse and visualization.
Dashboards are an essential part of data analysis and are used to track, analyze and visualize.
These Dashboard metadata model includes some basic fields like `urn` and `source`, etc and a list of `Chart`.
There are multiple dashboards that are essential for Data Analysis such as metabase, grafana, tableau, etc.
Please refer the list of Dashboards meteor currently supports [here](../reference/extractors.md).
Please refer to the list of Dashboards meteor currently supports [here](../reference/extractors.md).

- [Chart](https://github.com/odpf/proton/blob/main/odpf/assets/chart.proto):
Charts are included in all the Dashboard and are result of certain queries in a Dashboard.
Charts are included in all the Dashboard and are the result of certain queries in a Dashboard.
Information about them includes the information of the query and few similar details.

- [User](https://github.com/odpf/proton/blob/main/odpf/assets/user.proto):
This metadata model is used for defining the output of extraction on Users accounts.
Some of these source can be GitHub, Workday, Google Suite, LDAP.
Please refer the list of user meteor currently supports [here](../reference/extractors.md).
Some of these sources can be GitHub, Workday, Google Suite, LDAP.
Please refer to the list of user meteor currently supports [here](../reference/extractors.md).

- [Table](https://github.com/odpf/proton/blob/main/odpf/assets/table.proto):
This metadata model is being used by extractors based around `databases` or for the ones that store data in tabular format.
It contains various fields that includes `schema` of the table and other access related information.
It contains various fields that include `schema` of the table and other access related information.

- [Job](https://github.com/odpf/proton/blob/main/odpf/assets/job.proto):
Most of the data is being streamed as queues by kafka or other stack in DE pipeline.
And hence Job is a metadata model build for this purpose.
And hence Job is a metadata model built for this purpose.

`Proto` has been used to define these metadata models.
To check their implementation please refer [here](https://github.com/odpf/proton/tree/main/odpf/assets).
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/reference/sinks.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ sinks:

`compass`

Upload metadata to a given `type` in [Compass](https://github.com/odpf/meteor/tree/cb12c3ecf8904cf3f4ce365ca8981ccd132f35d0/docs/reference/github.com/odpf/compass/README.md). Request will be send via HTTP to given host.
Upload metadata to a given `type` in [Compass](https://github.com/odpf/meteor/tree/cb12c3ecf8904cf3f4ce365ca8981ccd132f35d0/docs/reference/github.com/odpf/compass/README.md). Request will be sent via HTTP to a given host.

### Sample usage of compass sink

Expand Down