Add index_type tsdb support to OpenSearch sink for Prometheus metrics#6691
Add index_type tsdb support to OpenSearch sink for Prometheus metrics#6691srikanthpadakanti wants to merge 6 commits intoopensearch-project:mainfrom
Conversation
Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>
72538da to
a0887ed
Compare
…e placement Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>
a0887ed to
e7b03a9
Compare
✅ License Header Check PassedAll newly added files have proper license headers. Great work! 🎉 |
Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>
|
@dlvenable @kkondaka Could you please trigger the remaining workflows and review this? |
dlvenable
left a comment
There was a problem hiding this comment.
Thank you @srikanthpadakanti for this contribution!
...nsearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/OpenSearchSink.java
Outdated
Show resolved
Hide resolved
.../main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/TSDBDocumentBuilder.java
Outdated
Show resolved
Hide resolved
...t/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/TSDBDocumentBuilderTest.java
Outdated
Show resolved
Hide resolved
.../main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/TSDBDocumentBuilder.java
Outdated
Show resolved
Hide resolved
...t/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/TSDBDocumentBuilderTest.java
Show resolved
Hide resolved
...nsearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/OpenSearchSink.java
Outdated
Show resolved
Hide resolved
.../main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexManagerFactory.java
Outdated
Show resolved
Hide resolved
...rationTest/java/org/opensearch/dataprepper/plugins/sink/opensearch/OpenSearchSinkTsdbIT.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>
dlvenable
left a comment
There was a problem hiding this comment.
Thank you @srikanthpadakanti for this contribution!
|
@kkondaka , Could you take a review of this as you have worked quite a bit with Prometheus? |
| timestamp, sum != null ? sum : 0.0)); | ||
| } | ||
|
|
||
| private static String[][] sortAndSanitizeAttributes(final Map<String, Object> attributes) { |
There was a problem hiding this comment.
Looks like some of this code is similar to what is in the Prometheus sink. Should we move this code to some common library?
There was a problem hiding this comment.
The sanitization logic here serves a different purpose. PrometheusTimeSeries sanitizes for Prometheus remote write naming conventions (unit suffixes, OpenMetrics rules), while TSDBDocumentBuilder sanitizes for the TSDB space-delimited keyword format (spaces to underscores, key prefix rules).
The methods share a similar name but the actual logic does not overlap. Extracting to a common library would couple two different output format requirements.
There was a problem hiding this comment.
@srikanthpadakanti , On this topic, does Prometheus have any Java libraries that might have similar logic?
There was a problem hiding this comment.
@dlvenable Checked PrometheusNaming in prometheus-metrics-model. It validates and escapes names for Prometheus exposition formats but does not do the kind of transformation we need. Their sanitizeLabelName only strips leading double-underscore prefixes, it does not replace invalid characters. They have no equivalent for replacing spaces in values (our TSDB delimiter requirement) or building sorted key-value label strings. The methods solve different problems at different layers.
Description
Add
index_type: tsdbsupport to the OpenSearch sink for writing Prometheus metrics to OpenSearch TSDB indices. The sink converts Data Prepper Metric events (Gauge, Sum, Histogram, Summary) into TSDB's{labels, timestamp, value}document format, expanding complex types (histograms into per-bucket documents, summaries into per-quantile documents) so no intermediate processor is needed.Issues Resolved
#6644
Resolves #6644
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.