Feature Request: Prometheus Remote Write v1 Source for OpenSearch TSDB

[srikanthpadakanti]

Feature Request: Prometheus Remote Write v1 Source for Data Prepper

Is your feature request related to a problem?

Currently, there is no native way to ingest metrics from Prometheus servers directly into OpenSearch. Organizations using Prometheus for metrics collection must build and maintain custom exporters to convert Prometheus Remote Write format, which is time-consuming and error-prone.

What solution would you like?

Implement a Prometheus Remote Write v1.0 source plugin for Data Prepper that:

• Receives Prometheus Remote Write requests on a configurable HTTP endpoint (default: http://data-prepper:9090/api/v1/write)
• Parses the protocol (Snappy decompression + Protocol Buffer parsing)
• Converts to multiple output formats (TSDB, OTEL, OpenSearch, Prometheus)
• Supports OTLP collectors via OpenTelemetry Collector's prometheusremotewrite exporter

Configuration:

Prometheus side (2-line change):

# prometheus.yml
remote_write:
  - url: "http://data-prepper:9090/api/v1/write"

Data Prepper pipeline:

prometheus-pipeline:
    source:
      prometheus:
        port: 9090
        path: "/api/v1/write"
        # Source emits Metric events (Data Prepper internal model)

    sink:
      - opensearch:
          hosts: ["https://opensearch:9200"]
          index_type: tsdb  # Sink converts Metric → TSDB format
          index: metrics

Format Specifications

TSDB Format

OpenSearch TSDB expects documents with space-separated labels format:

Format:

{
  "labels": "__name__ http_requests_total method POST handler /api/items status 200",
  "timestamp": 1633072800000,
  "value": 1.1
}

Specification:
- https://github.com/opensearch-project/time-series-db
- Labels are space-separated key-value pairs: key1 value1 key2 value2
- See https://github.com/opensearch-project/time-series-db#index-some-metrics

Sink Configuration:
sink:
  - opensearch:
      hosts: ["https://opensearch:9200"]
      index_type: tsdb
      index: metrics

OTEL Format

OpenTelemetry Metrics format following the OTEL specification:

Format:
{
  "kind": "gauge",
  "name": "http_requests",
  "value": 100,
  "attributes": {"method": "GET"},
  "time": "2024-02-12T15:30:00.000Z"
}

Specification:
- https://opentelemetry.io/docs/specs/otel/metrics/data-model/
- Data Prepper's internal Metric interface aligns with OTEL model

Sink Configuration:
sink:
  - opensearch:
      hosts: ["https://opensearch:9200"]
      index_type: otel_metrics  # If supported
      index: metrics

OpenSearch Format

Standard OpenSearch document format with conventional field names:

Format:
{
  "@timestamp": "2024-02-12T15:30:00Z",
  "metric_name": "http_requests",
  "metric_value": 100,
  "labels": {"method": "GET"}
}

Specification:
- Standard OpenSearch document structure
- Compatible with existing dashboards and queries

Sink Configuration:
sink:
  - opensearch:
      hosts: ["https://opensearch:9200"]
      index: metrics  # Regular index

Prometheus Format

Preserve original Prometheus Remote Write structure for compatibility:

Format:
{
  "name": "http_requests",
  "labels": {
    "__name__": "http_requests",
    "method": "GET"
  },
  "value": 100,
  "timestamp": 1707523200000
}

Specification:
- https://prometheus.io/docs/specs/remote_write_spec/
- Used for debugging or forwarding to other Prometheus systems

Sink Configuration:
sink:
  - prometheus:
      url: "https://prometheus:9200"  # Amazon Managed Prometheus

Examples

Basic Example

Input: Prometheus Remote Write

TimeSeries {
  labels: [{name: "__name__", value: "http_requests"}, {name: "method", value: "GET"}]
  samples: [{value: 100, timestamp: 1707523200000}]
}

Output (TSDB format):

{"labels": "__name__ http_requests method GET", "timestamp": 1707523200000, "value": 100}

Output (OTEL format):

{"kind": "gauge", "name": "http_requests", "value": 100, "attributes": {"method": "GET"}}

Output (OpenSearch format):

{"@timestamp": "2024-02-12T15:30:00Z", "metric_name": "http_requests", "metric_value": 100, "labels": {"method": "GET"}}

Output (Prometheus format):

{"name": "http_requests", "labels": {"__name__": "http_requests", "method": "GET"}, "value": 100, "timestamp": 1707523200000}

  ## Multi-Sink Example

  Send to both Prometheus and OpenSearch TSDB simultaneously:

  ```yaml
  prometheus-pipeline:
    source:
      prometheus:
        port: 9090
        path: "/api/v1/write"
    sink:
      - prometheus:
          url: "https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-xxx"
      - opensearch:
          hosts: ["https://opensearch:9200"]
          index_type: tsdb
          index: metrics

[dlvenable]

Thank you @srikanthpadakanti for opening this issue and contributing toward the project.

Data Prepper has an internal event model for data in the pipeline. We use this model to maintain compatibility between the plugins. In addition to our Event model, we have some predefined specialized types. One of these is the Metric type. This new prometheus source should emit Data Prepper events that implement the Metric interface.

Right now, the prometheus sink can send Metric events to Prometheus. So by sending Metric events, from this source you could write those same events back to Prometheus without any additional processors.

prometheus-pipeline:
  source:
    prometheus:
      port: 9090
      path: "/api/v1/write"
  sink:
    - prometheus:
        url: "https://prometheus:9200"

(The Prometheus sink only supports Amazon Managed Prometheus presently, but once it supports open-source this should be valid).

Now, what about writing to other sinks? I believe you want to write to the new OpenSearch-TSDB. I'm not familiar with it, but it looks like it accepts normal OpenSearch documents that come with a certain mapping. Data Prepper already supports similar scenarios in the opensearch sink for trace data. The sink will send documents using mappings that work with the observability plugin. You can see this example for how we support it.

As with the prometheus sink we should be able to write to OpenSearch using the opensearch sink, ideally without processors. Here is a possible approach to how this could work (again, I'm not familiar with TSDB, so perhaps more is needed):

prometheus-pipeline:
  source:
    prometheus:
      port: 9090
      path: "/api/v1/write"
  sink:
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        index_type: tsdb
        index: metrics

One advantage to avoid processors is that we can also send to both sinks in one pipeline.

prometheus-pipeline:
  source:
    prometheus:
      port: 9090
      path: "/api/v1/write"
  sink:
    - prometheus:
        url: "https://prometheus:9200"
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        index_type: tsdb
        index: metrics

In your description above you mention a few types to support - TSDB, OTEL, OpenSearch, Prometheus. Can you clarify what each of these formats is and include a link to the spec or documentation for each? That should help us better define the requirement for the source.

[mishail]

In your description above you mention a few types to support - TSDB, OTEL, OpenSearch, Prometheus. Can you clarify what each of these formats is and include a link to the spec or documentation for each?

I suggest we focus on the "source" side of the things in this issue. The new source shall be able to accept data in Prometheus Remote Write v1 format and convert them to Metric.

Transformation shall be handled by processors, and those can be added later when needed.

[srikanthpadakanti]

Thank you @dlvenable
I understand now that the source should emit Data Prepper Metric events (internal model), and the sink handles format conversion via index_type. This is much cleaner than using a processor. I've updated the feature request to clarify the format specifications:

Questions:

1. I think opensearch index_type: tsdb, is not supported as of now and this need to be added. Would that scope be with this feature request? or you want me to raise another issue - to keep it clean?
2. Should the source default to emitting OTEL-format Metric events, similar to how otel_trace_source emits trace events?

[srikanthpadakanti]

Hello @dlvenable Please take a look into the PR #6627
Thanks.