Skip to content

feat: add HTTP extractor #442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sudo-suhas merged 1 commit into from
Jan 11, 2023
Merged

feat: add HTTP extractor #442

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 41 additions & 2 deletions docs/docs/reference/extractors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# Extractors

Meteor currently supports metadata extraction on these data sources. To perform extraction on any of these you need to create a recipe file with instructions as mentioned [here](../concepts/recipe.md). In the `sample-recipe.yaml` add `source` information such as `type` from the table below and `config` for that particular extractor can be found by visiting the link in the `type` field.
Meteor currently supports metadata extraction on these data sources. To perform
extraction on any of these you need to create a recipe file with instructions as
mentioned [here](../concepts/recipe.md). In the `sample-recipe.yaml`
add `source` information such as `type` from the table below and `config` for
that particular extractor can be found by visiting the link in the `type` field.

## Extractors Feature Matrix

Expand Down Expand Up @@ -31,7 +35,6 @@ Meteor currently supports metadata extraction on these data sources. To perform
| [`tableau`][tableau-readme] | ✅ | ✅ | ✅ | ✗ | ✗ |
| [`redash`][redash-readme] | ✅ | ✗ | ✗ | ✗ | ✗ |


### Topic

| Type | Profile | Schema | Ownership | Lineage | Tags | Custom |
Expand Down Expand Up @@ -76,31 +79,67 @@ Meteor currently supports metadata extraction on these data sources. To perform
|:--------------------------|:----------|:----------|:------------|:-------|
| [`merlin`][merlin-readme] | ✅ | ✅ | ✗ | ✅ | ✅ |

### Generic

These are special type of extractors that are capable of extracting _any_ type
of asset.

| Type | Ownership | Upstreams | Downstreams | Custom |
|:----------------------|:----------|:----------|:------------|:-------|
| [`http`][http-readme] | ✅ | ✅ | ✅ | ✅ | ✅ |

<!--- Not using relative links because that breaks the docs build -->

[clickhouse-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/clickhouse/README.md

[couchdb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/couchdb/README.md

[mongodb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mongodb/README.md

[mssql-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mssql/README.md

[mysql-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mysql/README.md

[postgres-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/postgres/README.md

[cassandra-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/cassandra/README.md

[oracle-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/oracle/README.md

[mariadb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mariadb/README.md

[redshift-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/redshift/README.md

[presto-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/presto/README.md

[snowflake-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/snowflake/README.md

[grafana-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/grafana/README.md

[metabase-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/metabase/README.md

[superset-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/superset/README.md

[tableau-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/tableau/README.md

[redash-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/redash/README.md

[kafka-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/kafka/README.md

[github-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/github/README.md

[shield-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/shield/README.md

[gsuite-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/gsuite/README.md

[gcs-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/gcs/README.md

[optimus-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/optimus/README.md

[caramlstore-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/caramlstore/README.md

[application-yaml-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/application_yaml/README.md

[merlin-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/merlin/README.md

[http-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/http/README.md
4 changes: 4 additions & 0 deletions plugins/base_plugin_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ func TestBasePluginValidate(t *testing.T) {
invalidConfig := struct {
FieldA string `mapstructure:"field_a" validate:"required"`
FieldB string `mapstructure:"field_b" validate:"url"`
Nested struct {
FieldC string `mapstructure:"field_c" validate:"required"`
} `mapstructure:"nested"`
}{}

basePlugin := plugins.NewBasePlugin(plugins.Info{}, &invalidConfig)
Expand All @@ -66,6 +69,7 @@ func TestBasePluginValidate(t *testing.T) {
Errors: []plugins.ConfigError{
{Key: "field_a", Message: "validation for field 'field_a' failed on the 'required' tag"},
{Key: "field_b", Message: "validation for field 'field_b' failed on the 'url' tag"},
{Key: "nested.field_c", Message: "validation for field 'nested.field_c' failed on the 'required' tag"},
},
})
})
Expand Down
2 changes: 1 addition & 1 deletion plugins/extractors/application_yaml/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ source:
| Key | Value | Example | Description | Required? |
|:-------------|:---------|:------------------|:-----------------------------------------------------------------------------------------------------------------------|:----------|
| `file` | `string` | `meteor.app.yaml` | File path of `application.yaml` | ✅ |
| `env_prefix` | `string` | `CI` | Prefix for environment variables. These are made available as variables in `application.yaml` with the prefix trimmed. | ❌ |
| `env_prefix` | `string` | `CI` | Prefix for environment variables. These are made available as variables in `application.yaml` with the prefix trimmed. | ✘ |

### `application.yaml` format

Expand Down
9 changes: 0 additions & 9 deletions plugins/extractors/application_yaml/application_yaml.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,15 +73,6 @@ func New(logger log.Logger) *Extractor {
return &e
}

// Init initializes the extractor
func (e *Extractor) Init(ctx context.Context, config plugins.Config) error {
if err := e.BaseExtractor.Init(ctx, config); err != nil {
return err
}

return nil
}

func (e *Extractor) Extract(_ context.Context, emit plugins.Emit) error {
tmpl, err := template.ParseFiles(e.config.File)
if err != nil {
Expand Down
10 changes: 5 additions & 5 deletions plugins/extractors/caramlstore/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ source:

## Inputs

| Key | Value | Example | Description | Required? |
|:------------------|:---------|:----------------------|:----------------------------------------------------------|:----------|
| `url` | `string` | `caraml-store.com:80` | caraml-store's host URL | ✅ |
| `max_size_in_mb` | `int` | `10` | Max MB for gRPC client to receive message. Default is 45. | ❌ |
| `request_timeout` | `string` | `10s` | Timeout for gRPC requests to caraml-store | ❌ |
| Key | Value | Example | Description | Required? |
|:------------------|:---------|:----------------------|:-----------------------------------------------------------|:----------|
| `url` | `string` | `caraml-store.com:80` | caraml-store's host URL | ✅ |
| `max_size_in_mb` | `int` | `10` | Max MB for gRPC client to receive message. Default is 45. | ✘ |
| `request_timeout` | `string` | `10s` | Timeout for gRPC requests to caraml-store. Default is 10s. | ✘ |

## Outputs

Expand Down
Loading