Skip to content

Add index_type tsdb support to OpenSearch sink for Prometheus metrics#6691

Open
srikanthpadakanti wants to merge 6 commits intoopensearch-project:mainfrom
srikanthpadakanti:opensearch-sink-tsdb-6644
Open

Add index_type tsdb support to OpenSearch sink for Prometheus metrics#6691
srikanthpadakanti wants to merge 6 commits intoopensearch-project:mainfrom
srikanthpadakanti:opensearch-sink-tsdb-6644

Conversation

@srikanthpadakanti
Copy link
Copy Markdown
Contributor

Description

Add index_type: tsdb support to the OpenSearch sink for writing Prometheus metrics to OpenSearch TSDB indices. The sink converts Data Prepper Metric events (Gauge, Sum, Histogram, Summary) into TSDB's {labels, timestamp, value} document format, expanding complex types (histograms into per-bucket documents, summaries into per-quantile documents) so no intermediate processor is needed.

Issues Resolved

#6644
Resolves #6644

Check List

  • [ X ] New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • [ X ] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>
@srikanthpadakanti srikanthpadakanti force-pushed the opensearch-sink-tsdb-6644 branch 2 times, most recently from 72538da to a0887ed Compare March 31, 2026 21:38
…e placement

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>
@srikanthpadakanti srikanthpadakanti force-pushed the opensearch-sink-tsdb-6644 branch from a0887ed to e7b03a9 Compare March 31, 2026 21:40
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 1, 2026

✅ License Header Check Passed

All newly added files have proper license headers. Great work! 🎉

Srikanth Padakanti added 2 commits April 1, 2026 11:25
Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>
@srikanthpadakanti
Copy link
Copy Markdown
Contributor Author

srikanthpadakanti commented Apr 1, 2026

@dlvenable @kkondaka Could you please trigger the remaining workflows and review this?

Copy link
Copy Markdown
Member

@dlvenable dlvenable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @srikanthpadakanti for this contribution!

Srikanth Padakanti added 2 commits April 9, 2026 11:02
Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>
Copy link
Copy Markdown
Member

@dlvenable dlvenable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @srikanthpadakanti for this contribution!

@dlvenable
Copy link
Copy Markdown
Member

@kkondaka , Could you take a review of this as you have worked quite a bit with Prometheus?

timestamp, sum != null ? sum : 0.0));
}

private static String[][] sortAndSanitizeAttributes(final Map<String, Object> attributes) {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like some of this code is similar to what is in the Prometheus sink. Should we move this code to some common library?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sanitization logic here serves a different purpose. PrometheusTimeSeries sanitizes for Prometheus remote write naming conventions (unit suffixes, OpenMetrics rules), while TSDBDocumentBuilder sanitizes for the TSDB space-delimited keyword format (spaces to underscores, key prefix rules).

The methods share a similar name but the actual logic does not overlap. Extracting to a common library would couple two different output format requirements.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srikanthpadakanti , On this topic, does Prometheus have any Java libraries that might have similar logic?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlvenable Checked PrometheusNaming in prometheus-metrics-model. It validates and escapes names for Prometheus exposition formats but does not do the kind of transformation we need. Their sanitizeLabelName only strips leading double-underscore prefixes, it does not replace invalid characters. They have no equivalent for replacing spaces in values (our TSDB delimiter requirement) or building sorted key-value label strings. The methods solve different problems at different layers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: OpenSearch Sink index_type: tsdb for TSDB Index Support

3 participants