Collector's Internal Metrics Not Disabled When Specifying service:telemetry:metrics:level:none

[RaginiDhatric]

Describe the bug

The setting of service: telemetry: metrics: level: none successfully disables basic and normal-level metrics in the OpenTelemetry Collector; however, detailed-level metrics continue to be exported to Stackdriver, Prometheus and managed Prometheus metrics exporters.

Steps to reproduce

1. Set service:telemetry:metrics:level:none in the OpenTelemetry Collector configuration.
2. Deploy the Collector with this configuration.
3. Observe the metrics exported to Google Stackdriver, managed Prometheus and Prometheus noting that detailed-level internal metrics are still being collected and exported.

What did you expect to see?

All internal metrics, including detailed-level metrics, should not be collected or exported

What did you see instead?

Detailed-level metrics are still being collected and exported. Example below

What version did you use?

v0.99.0

What config did you use?

(edited by @mx-psi for clarity)

exporters:
googlecloud:
  metric:
    service_resource_labels: false
    instrumentation_library_labels: false
  prometheus:
    endpoint: 0.0.0.0:9090
    metric_expiration: 1m
    add_metric_suffixes: false
  googlemanagedprometheus:
    metric:
      add_metric_suffixes: false
      extra_metrics_config:
        enable_scope_info: false
        enable_target_info: false
    project: test
  logging: {}
processors:
  batch:
    send_batch_size: 200
    timeout: 0s
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
  resourcedetection:
    detectors:
    - gcp
    timeout: 10s
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317
      http:
        endpoint: ${env:MY_POD_IP}:4318
  zipkin:
     endpoint: 0.0.0.0:9411
service:
  pipelines:
    logs:
      exporters:
      - logging
      processors:
      - memory_limiter
      - batch
      receivers:
      - otlp
    metrics:
      exporters:
      - googlecloud
      - googlemanagedprometheus
      - prometheus
      processors:
      - resourcedetection
      - memory_limiter
      - batch
      receivers:
      - otlp
    traces:
      exporters:
        - otlp
       processors:
        - memory_limiter
        - batch
       receivers:
        - zipkin
        - otlp
  telemetry:
    metrics:
      level: none
    logs:
      encoding: json

Environment

Additional context

Initially by setting metrics:level:none option disabled basic level and normal level metrics listed here but i still continue to see detailed level metrics.
Supported logs:

[mx-psi]

@RaginiDhatric Can you try with the latest version (v0.106.0)?

[codeboten]

It looks like the source of these metrics are from the googlecloud exporter? Maybe @dashpole knows more here

[dashpole]

I suspect these are opencensus metrics. Is the OC bridge still used when metrics level is set to none?

[RaginiDhatric]

@mx-psi okay, let me try and get back!!

@codeboten I see these metrics from all the exporters i configured i.e googlecloud , googlemanagedprometheus and prometheus.

@dashpole I dont think so, because googlecloud exporter exports it as custom.googleapis.com/opentelemetry/rpc.client.duration and googlemanagedprometheus exports as prometheus.googleapis.com/rpc_client_duration/histogram

[dashpole]

nevermind. OC grpc metrics have a different name: https://github.com/census-instrumentation/opencensus-go/blob/01e6da5fc01c42aca1e0ce315f41744876e9fcbb/plugin/ocgrpc/server_metrics.go#L16

[RaginiDhatric]

@mx-psi I upgraded to the latest version (v0.106.1) but it did not help in disabling internal metrics for any of the exporters. I still see the following metrics when API calls are made i.e rpc_client_duration, rpc_client_request_size, rpc_client_requests_per_rpc, rpc_client_response_size, rpc_client_responses_per_rpc.

[atib92]

I am also observing the same issue. Irrespective of the 'level' value set via config, it seems the collector is configured with level=Normal

2024-08-05T16:36:15.305Z  info  service@v0.106.1/telemetry.go:96  Serving metrics {"address": ":8888", "metrics level": "Normal"}

[mx-psi]

@RaginiDhatric @atib92 It would be helpful if you can test which of the components is generating the metrics. To do this you can replace the different components and check if the metrics are still seen.

[atib92]

Mine turned out to be a false positive. The otel configs were not correctly mounted in the container which was causing the default configs to get loaded which probably enables debug metrics.

[RaginiDhatric]

@mx-psi I tested the setup by simplifying the OpenTelemetry Collector configuration to primarily include only the metrics exporter component. I still observed that internal telemetry metrics were being generated and exported. Below, I have detailed the testing configuration and my findings:

exporters:
  googlemanagedprometheus:
    metric:
      add_metric_suffixes: false
      extra_metrics_config:
        enable_scope_info: false
        enable_target_info: false
    project: gcp-project-name
  prometheus:
    add_metric_suffixes: false
    endpoint: 0.0.0.0:9090
    metric_expiration: 1m
extensions:
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
processors:
  batch: {}
  k8sattributes: null
  k8sattributes/2:
    extract:
      labels:
      - from: pod
        key: app
        tag_name: container_name
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
  resourcedetection:
    detectors:
    - gcp
    timeout: 10s
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317
      http:
        endpoint: ${env:MY_POD_IP}:4318
service:
  extensions:
  - health_check
  pipelines:
    metrics:
      exporters:
      - prometheus
      - googlemanagedprometheus
      processors:
      - resourcedetection
      - k8sattributes
      - memory_limiter
      - batch
      receivers:
      - otlp
  telemetry:
    logs:
      encoding: json
    metrics:
      level: none

Since I excluded the Google Cloud exporter from my configuration, I observed that the internal telemetry metrics were no longer exported through that particular exporter. However, these metrics were still visible for both the Google Managed Prometheus and Prometheus exporters.

Metrics i observed:
TYPE rpc_client_duration histogram
TYPE rpc_client_request_size histogram
TYPE rpc_client_requests_per_rpc histogram
TYPE rpc_client_response_size histogram
TYPE rpc_client_responses_per_rpc histogram

Detailed logs:

[adriangb]

I also have not been able to get any config to disable internal telemetry. This should be documented (and work).

[codeboten]

@adriangb can you share the configuration you've used? I just tested the following:

service:
  telemetry:
    metrics:
      level: detailed

curl localhost:8888/metrics
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds [alpha]
# TYPE otelcol_process_cpu_seconds counter
otelcol_process_cpu_seconds{service_instance_id="582f2176-3448-4bdb-8032-616baabfc003",service_name="otelcontribcol",service_version="0.120.1-dev"} 0.15523283333333332
# HELP otelcol_process_memory_rss Total physical memory (resident set size) [alpha]
# TYPE otelcol_process_memory_rss gauge
...

then changed it to:

service:
  telemetry:
    metrics:
      level: none

curl localhost:8888/metrics
curl: (7) Failed to connect to localhost port 8888 after 0 ms: Couldn't connect to server

The internal metrics are no longer exported.