Skip to content

Commit 2a2143b

Browse files
sudo-suhassudo-suhas
authored
feat: add HTTP extractor (#442)
- Add HTTP extractor, a generic extractor that uses a script capable of emitting any type of asset. - Fix config error field key to support nested fields. - Disallow import of os module from scripts. - Remove unnecessary init method on application extractor. - Add test helper to assert slice of protos.
1 parent cdfd99c commit 2a2143b

17 files changed

Lines changed: 1584 additions & 25 deletions

File tree

‎docs/docs/reference/extractors.md‎

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
# Extractors
22

3-
Meteor currently supports metadata extraction on these data sources. To perform extraction on any of these you need to create a recipe file with instructions as mentioned [here](../concepts/recipe.md). In the `sample-recipe.yaml` add `source` information such as `type` from the table below and `config` for that particular extractor can be found by visiting the link in the `type` field.
3+
Meteor currently supports metadata extraction on these data sources. To perform
4+
extraction on any of these you need to create a recipe file with instructions as
5+
mentioned [here](../concepts/recipe.md). In the `sample-recipe.yaml`
6+
add `source` information such as `type` from the table below and `config` for
7+
that particular extractor can be found by visiting the link in the `type` field.
48

59
## Extractors Feature Matrix
610

@@ -31,7 +35,6 @@ Meteor currently supports metadata extraction on these data sources. To perform
3135
| [`tableau`][tableau-readme] | ✅ | ✅ | ✅ | ✗ | ✗ |
3236
| [`redash`][redash-readme] | ✅ | ✗ | ✗ | ✗ | ✗ |
3337

34-
3538
### Topic
3639

3740
| Type | Profile | Schema | Ownership | Lineage | Tags | Custom |
@@ -76,31 +79,67 @@ Meteor currently supports metadata extraction on these data sources. To perform
7679
|:--------------------------|:----------|:----------|:------------|:-------|
7780
| [`merlin`][merlin-readme] | ✅ | ✅ | ✗ | ✅ | ✅ |
7881

82+
### Generic
83+
84+
These are special type of extractors that are capable of extracting _any_ type
85+
of asset.
86+
87+
| Type | Ownership | Upstreams | Downstreams | Custom |
88+
|:----------------------|:----------|:----------|:------------|:-------|
89+
| [`http`][http-readme] | ✅ | ✅ | ✅ | ✅ | ✅ |
90+
7991
<!--- Not using relative links because that breaks the docs build -->
8092

8193
[clickhouse-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/clickhouse/README.md
94+
8295
[couchdb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/couchdb/README.md
96+
8397
[mongodb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mongodb/README.md
98+
8499
[mssql-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mssql/README.md
100+
85101
[mysql-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mysql/README.md
102+
86103
[postgres-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/postgres/README.md
104+
87105
[cassandra-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/cassandra/README.md
106+
88107
[oracle-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/oracle/README.md
108+
89109
[mariadb-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/mariadb/README.md
110+
90111
[redshift-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/redshift/README.md
112+
91113
[presto-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/presto/README.md
114+
92115
[snowflake-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/snowflake/README.md
116+
93117
[grafana-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/grafana/README.md
118+
94119
[metabase-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/metabase/README.md
120+
95121
[superset-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/superset/README.md
122+
96123
[tableau-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/tableau/README.md
124+
97125
[redash-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/redash/README.md
126+
98127
[kafka-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/kafka/README.md
128+
99129
[github-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/github/README.md
130+
100131
[shield-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/shield/README.md
132+
101133
[gsuite-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/gsuite/README.md
134+
102135
[gcs-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/gcs/README.md
136+
103137
[optimus-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/optimus/README.md
138+
104139
[caramlstore-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/caramlstore/README.md
140+
105141
[application-yaml-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/application_yaml/README.md
142+
106143
[merlin-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/merlin/README.md
144+
145+
[http-readme]: https://github.com/odpf/meteor/tree/main/plugins/extractors/http/README.md

‎plugins/base_plugin_test.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,9 @@ func TestBasePluginValidate(t *testing.T) {
5454
invalidConfig := struct {
5555
FieldA string `mapstructure:"field_a" validate:"required"`
5656
FieldB string `mapstructure:"field_b" validate:"url"`
57+
Nested struct {
58+
FieldC string `mapstructure:"field_c" validate:"required"`
59+
} `mapstructure:"nested"`
5760
}{}
5861

5962
basePlugin := plugins.NewBasePlugin(plugins.Info{}, &invalidConfig)
@@ -66,6 +69,7 @@ func TestBasePluginValidate(t *testing.T) {
6669
Errors: []plugins.ConfigError{
6770
{Key: "field_a", Message: "validation for field 'field_a' failed on the 'required' tag"},
6871
{Key: "field_b", Message: "validation for field 'field_b' failed on the 'url' tag"},
72+
{Key: "nested.field_c", Message: "validation for field 'nested.field_c' failed on the 'required' tag"},
6973
},
7074
})
7175
})

‎plugins/extractors/application_yaml/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ source:
1818
| Key | Value | Example | Description | Required? |
1919
|:-------------|:---------|:------------------|:-----------------------------------------------------------------------------------------------------------------------|:----------|
2020
| `file` | `string` | `meteor.app.yaml` | File path of `application.yaml` | ✅ |
21-
| `env_prefix` | `string` | `CI` | Prefix for environment variables. These are made available as variables in `application.yaml` with the prefix trimmed. | ❌ |
21+
| `env_prefix` | `string` | `CI` | Prefix for environment variables. These are made available as variables in `application.yaml` with the prefix trimmed. | ✘ |
2222

2323
### `application.yaml` format
2424

‎plugins/extractors/application_yaml/application_yaml.go‎

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -73,15 +73,6 @@ func New(logger log.Logger) *Extractor {
7373
return &e
7474
}
7575

76-
// Init initializes the extractor
77-
func (e *Extractor) Init(ctx context.Context, config plugins.Config) error {
78-
if err := e.BaseExtractor.Init(ctx, config); err != nil {
79-
return err
80-
}
81-
82-
return nil
83-
}
84-
8576
func (e *Extractor) Extract(_ context.Context, emit plugins.Emit) error {
8677
tmpl, err := template.ParseFiles(e.config.File)
8778
if err != nil {

‎plugins/extractors/caramlstore/README.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,11 @@ source:
1919
2020
## Inputs
2121
22-
| Key | Value | Example | Description | Required? |
23-
|:------------------|:---------|:----------------------|:----------------------------------------------------------|:----------|
24-
| `url` | `string` | `caraml-store.com:80` | caraml-store's host URL | ✅ |
25-
| `max_size_in_mb` | `int` | `10` | Max MB for gRPC client to receive message. Default is 45. | ❌ |
26-
| `request_timeout` | `string` | `10s` | Timeout for gRPC requests to caraml-store | ❌ |
22+
| Key | Value | Example | Description | Required? |
23+
|:------------------|:---------|:----------------------|:-----------------------------------------------------------|:----------|
24+
| `url` | `string` | `caraml-store.com:80` | caraml-store's host URL | ✅ |
25+
| `max_size_in_mb` | `int` | `10` | Max MB for gRPC client to receive message. Default is 45. | ✘ |
26+
| `request_timeout` | `string` | `10s` | Timeout for gRPC requests to caraml-store. Default is 10s. | ✘ |
2727

2828
## Outputs
2929

0 commit comments

Comments
 (0)