Remote write compliance: TestRemoteWrite/otelcollector/RepeatedLabels

[rakyll]

The OpenTelemetry collector is not passing https://github.com/prometheus/compliance/tree/main/remote_write TestRemoteWrite/otelcollector/RepeatedLabels.

=== CONT  TestRemoteWrite/otelcollector/RepeatedLabels
    labels.go:58:
        	Error Trace:	labels.go:58
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/RepeatedLabels
        	Messages:   	found zero samples for up{job="test"} = 0

[rakyll]

Related to #41.

[Aneurysm9]

This continues to fail even with the up metric present:

=== CONT  TestRemoteWrite/otelcollector/RepeatedLabels
    labels.go:58:
                Error Trace:    labels.go:58
                                                        main_test.go:105
                                                        main_test.go:69
                Error:          Should be true
                Test:           TestRemoteWrite/otelcollector/RepeatedLabels
                Messages:       found samples for test{a="1"}, none expected

[odeke-em]

Please assign this issue to me

[alolita]

All yours @odeke-em

[odeke-em]

I think the Prometheus Remote Write compliance tests are wrong. I have written a sample test and used Prometheus with it and Prometheus is able to successfully get the datapoints.

main.go

package main

import (
	"context"
	"fmt"
	"net/http"
	"os"
	"os/signal"
	"sync/atomic"
	"time"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	i := uint64(1)

	go func() {
		for {
			select {
			case <-ctx.Done():
				return
			case <-time.After(27 * time.Second):
				atomic.AddUint64(&i, 1)
			}
		}
	}()

	handler := func(rw http.ResponseWriter, req *http.Request) {
		println("Scraped", time.Now().Unix())
		fmt.Fprintf(rw, `
# HELP test A gauge
# TYPE test gauge
test{a="1",a="1"} %.1f`, float64(atomic.LoadUint64(&i)))
	}

	sigCh := make(chan os.Signal, 1)
	signal.Notify(sigCh, os.Interrupt)

	defer cancel()

	go func() {
		if err := http.ListenAndServe(":8855", http.HandlerFunc(handler)); err != nil {
			panic(err)
		}
	}()
	<-sigCh
}

Run it by go run main.go

prom.yaml

global:
  scrape_interval: 10s

  external_labels:
    monitor: 'otlc'

remote_write:

scrape_configs:
  - job_name: 'otlc'

    scrape_interval: 10s

    honor_labels: true

    static_configs:
        - targets: ['localhost:8855']

Run it by prometheus --config.file=prom.yaml --log.level=debug

Viewed on Prometheus

Synopsis

Prometheus itself doesn't reject these repeated labels, so why are the compliance tests rejecting it? Kindly cc-ing @tomwilkie @brian-brazil @rakyll @Aneurysm9 @alolita.

[brian-brazil]

Prometheus itself doesn't reject these repeated labels

If that's the case, then that's a bug in Prometheus. Are you sure you're using the latest version? We had to fix this at one point.

[roidelapluie]

We reject them, in TSDB:

https://github.com/prometheus/prometheus/blob/7bc11dcb06640ce22b4e15eb52b2c065f97cf79a/tsdb/head.go#L1374-L1376