Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion docs/docs/concepts/sink.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,23 @@ sinks:
path: "./dir/sample.yaml"
format: "yaml"
```

Sinks metadata to a file in `json/yaml` format as per the config defined.

* **Stencil**

```yaml
sinks:
name: stencil
config:
host: https://stencil.com
namespace_id: myNamespace
schema_id: mySchema
format: json
send_format_header: false
```

Upload metadata of a given schema `format` in the existing `namespace_id` present in Stencil. Request will be sent via HTTP to a given host.

## Upcoming sinks

* HTTP
Expand Down
17 changes: 17 additions & 0 deletions docs/docs/reference/sinks.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,23 @@ sinks:
format: "yaml"
```

## Stencil

`stencil`

Upload metadata of a given schema `format` in the existing `namespace_id` present in [Stencil](https://github.com/odpf/meteor/tree/cb12c3ecf8904cf3f4ce365ca8981ccd132f35d0/docs/reference/github.com/odpf/stencil/README.md). Request will be sent via HTTP to a given host.

```yaml
sinks:
name: stencil
config:
host: https://stencil.com
namespace_id: myNamespace
schema_id: mySchema
format: json
send_format_header: false
```

_**Notes**_

Compass' Type requires certain fields to be sent, hence why `mapping` config is needed to map value from any of our metadata models to any field name when sending to Compass. Supports getting value from nested fields.
1 change: 1 addition & 0 deletions plugins/sinks/populate.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ import (
_ "github.com/odpf/meteor/plugins/sinks/console"
_ "github.com/odpf/meteor/plugins/sinks/file"
_ "github.com/odpf/meteor/plugins/sinks/kafka"
_ "github.com/odpf/meteor/plugins/sinks/stencil"
)
32 changes: 32 additions & 0 deletions plugins/sinks/stencil/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Stencil

Stencil is a schema registry that provides schema management and validation dynamically, efficiently, and reliably to ensure data compatibility across applications.

## Usage

```yaml
sinks:
name: stencil
config:
host: https://stencil.com
namespace_id: test-namespace
schema_id: example
format: json
send_format_header: false
```

## Config Definition

| Key | Value | Example | Description | |
| :-- | :---- | :------ | :---------- | :-- |
|`host` | `string` | `https://stencil.com` | The hostname of the stencil service | *required*|
| `namespace_id` | `string` | `myNamespace` | The namespace ID of the stencil service | *required* |
|`schema_id` | `string` | `mySchmea` | The schema ID which will be created in the above-mentioned namespace | *required*|
|`format` | `string` | `json` | The schema format in which data will sink to stencil | *optional*|
|`send_format_header` | `bool` | `false` | If schema format needs to be changed. Suppose changing format from json to avro,
provide below config value as true and schema format in format config. | *optional*|


## Contributing

Refer to the contribution guidelines for information on contributing to this module.
50 changes: 50 additions & 0 deletions plugins/sinks/stencil/schemas.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
package stencil

type JsonType string
type AvroType string

const (
JsonTypeObject JsonType = "object"
JsonTypeString JsonType = "string"
JsonTypeNumber JsonType = "number"
JsonTypeArray JsonType = "array"
JsonTypeBoolean JsonType = "boolean"
JsonTypeNull JsonType = "null"

AvroTypeNull AvroType = "null"
AvroTypeBoolean AvroType = "boolean"
AvroTypeInteger AvroType = "int"
AvroTypeLong AvroType = "long"
AvroTypeFloat AvroType = "float"
AvroTypeDouble AvroType = "double"
AvroTypeBytes AvroType = "bytes"
AvroTypeString AvroType = "string"
AvroTypeRecord AvroType = "record"
AvroTypeArray AvroType = "array"
AvroTypeMap AvroType = "map"
)

type JsonSchema struct {
Id string `json:"$id"`
Schema string `json:"$schema"`
Title string `json:"title"`
Type JsonType `json:"type"`
Properties map[string]JsonProperty `json:"properties"`
}

type JsonProperty struct {
Type []JsonType `json:"type"`
Description string `json:"description"`
}

type AvroSchema struct {
Type string `json:"type"`
Namespace string `json:"namespace"`
Name string `json:"name"`
Fields []AvroFields `json:"fields"`
}

type AvroFields struct {
Name string `json:"name"`
Type interface{} `json:"type"`
}
Loading