-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Closed
Labels
domain: codecsAnything related to Vector's codecs (encoding/decoding)Anything related to Vector's codecs (encoding/decoding)domain: sinksAnything related to the Vector's sinksAnything related to the Vector's sinkstype: enhancementA value-adding code change that enhances its existing functionality.A value-adding code change that enhances its existing functionality.
Description
Like #9404 but for encoding/sinks
Issues that will be resolved with the integration:
Native encoding issues:
- Issues with parsing encoding configurations that prevent the user from specifying that there should be no framing (see review comments on enhancement(socket sink): Integrate
encoding::Encoderwithsocketsink #10684) - Sinks with configurable codecs should have input types related to the configured codec #12135
- Default framing based on encoding #12134
Other issues:
- Add sink octet framing that matches source octet framing #12434
- docs(codecs, sinks): Document changes to sink encoding #13331
Priority of sinks (check means "has open PR" or "marked as not applicable"):
-
socket -
http -
kafka -
datadog_archives -
datadog_events -
datadog_logs -
datadog_traces -
aws_s3 -
azure_blob -
gcp_cloudstorage -
gcp_pubsub -
gcp_stackdriver_logs -
console -
file -
vector -
aws_sqs -
elasticsearch -
splunk_hec_logs -
loki -
aws_cloudwatch_logs -
aws_kinesis_firehose -
aws_kinesis_streams -
azure_monitor_logs -
blackhole -
clickhouse -
honeycomb -
humio_logs -
influxdb -
logdna -
nats -
new_relic -
new_relic_logs -
papertrail -
pulsar -
redis -
sematext -
websocket
| Sink | Issue | PR | Status | Comment | Legacy encoding |
|---|---|---|---|---|---|
aws_cloudwatch_logs |
- | #12175 | Merged | - | EncodingConfig<StandardEncodings> |
aws_kinesis_firehose |
- | #12176 | Merged | - | EncodingConfig<StandardEncodings> |
aws_kinesis_streams |
- | #12177 | Merged | - | EncodingConfig<StandardEncodings> |
aws_s3 |
- | #12136 | Merged | - | EncodingConfig<StandardEncodings> |
aws_sqs |
- | #12550 | Merged | - | EncodingConfig<(Text,Json)> |
azure_blob |
- | #12179 | Merged | - | EncodingConfig<StandardEncodings> |
azure_monitor_logs |
- | Blocked | Needs improvements to Encoder to emit JSON value when serializer supports JSON. (Part of #12495.) |
EncodingConfigWithDefault<(Default)> |
|
blackhole |
- | - | Not applicable | - | - |
clickhouse |
- | - | Not applicable | Has opinionated fixed encoding | EncodingConfigWithDefault<(Default)> |
console |
- | #12181 | In review | EncodingConfig<StandardEncodings> |
|
datadog_archives |
- | - | Not applicable | Has opinionated fixed encoding | - |
datadog_events |
- | - | Not applicable | Has opinionated fixed encoding | - |
datadog_logs |
- | - | Not applicable | Has opinionated fixed encoding | - |
datadog_traces |
- | - | Not applicable | Has opinionated fixed encoding | - |
elasticsearch |
- | Needs clarification | The Elasticsearch Bulk API expects entries in JSON format. | EncodingConfigFixed<{doc_type,suppress_type_name}> |
|
file |
- | #12548 | Merged | - | EncodingConfig<(Text,Ndjson)> |
gcp_cloud_storage |
- | #12488 | Merged | - | EncodingConfig<StandardEncodings> |
gcp_pubsub |
- | #12718 | In review | EncodingConfigWithDefault<(Default)> |
|
gcp_stackdriver_logs |
- | #12719 | Blocked | Needs improvements to Encoder to emit JSON value when serializer supports JSON. (Part of #12495.) |
EncodingConfigWithDefault<(Default)> |
honeycomb |
- | #12764 | Blocked | Needs improvements to Encoder to emit JSON value when serializer supports JSON. (Part of #12495.) |
- |
http |
#11579 | #11647 | Merged | - | EncodingConfig<(Text,Ndjson,Json)> |
humio_logs |
- | #12495 | In review | EncodingConfig<(Json,Text)> |
|
influxdb |
- | #12583 | Merged | - | EncodingConfigWithDefault<(Default)> |
kafka |
- | #12133 | Merged | - | EncodingConfig<StandardEncodings> |
logdna |
- | EncodingConfigWithDefault<(Default)> |
|||
loki |
- | #12558 | Merged | - | EncodingConfig<(Json,Text,Logfmt)> |
nats |
- | #12586 | Merged | - | EncodingConfig<(Text,Json)> |
new_relic |
- | EncodingConfigFixed<(Default)> |
|||
new_relic_logs |
- | EncodingConfigWithDefault<(Json)> |
|||
papertrail |
- | #12589 | Merged | - | EncodingConfig<(Text,Json)> |
pulsar |
- | #13139 | In review | EncodingConfig<(Text,Json,Avro)> |
|
redis |
- | #12596 | Merged | - | EncodingConfig<(Text,Json)> |
sematext |
- | EncodingConfigFixed<{doc_type,suppress_type_name}> |
|||
socket |
- | #10684 | Merged | - | EncodingConfig<(Text,Json)> |
splunk_hec_logs |
- | #12495 | In review | EncodingConfig<(Json,Text)> |
|
vector |
- | - | Not applicable | - | - |
websocket |
- | #13054 | In review | - | EncodingConfig<StandardEncodings> |
Metadata
Metadata
Assignees
Labels
domain: codecsAnything related to Vector's codecs (encoding/decoding)Anything related to Vector's codecs (encoding/decoding)domain: sinksAnything related to the Vector's sinksAnything related to the Vector's sinkstype: enhancementA value-adding code change that enhances its existing functionality.A value-adding code change that enhances its existing functionality.