[receivercreator] fix annotation discovery config unmarshal (#43745)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

This PR fixes the issue described at
#43730
where nested configurations are not unmarshaled properly and hence they
cannot be loaded later by the components:

```
2025-10-22T14:11:01.323Z        info    [email protected]/observerhandler.go:206 starting receiver       {"resource": {"service.instance.id": "a282b5ed-2088-41ab-8b31-e4c38209791d", "service.name": "otelcol-contrib", "service.version": "0.138.0"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/49541bff-1b96-443d-a3fa-6190d23609a7_8080", "endpoint": "10.1.1.86:8080", "endpoint_id": "k8s_observer/49541bff-1b96-443d-a3fa-6190d23609a7/http-metrics(8080)", "config": {"config":{"scrape_configs":[{"job_name":"prometheus_metrics","metrics_path":"/metrics","scrape_interval":"10s","static_configs":[{"targets":["`endpoint`"]}]}]},"trim_metric_suffixes":true}}
2025-10-22T14:11:01.324Z        error   [email protected]/observerhandler.go:222 failed to start receiver        {"resource": {"service.instance.id": "a282b5ed-2088-41ab-8b31-e4c38209791d", "service.name": "otelcol-contrib", "service.version": "0.138.0"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "receiver": "prometheus/49541bff-1b96-443d-a3fa-6190d23609a7_8080", "error": "failed to load \"prometheus/49541bff-1b96-443d-a3fa-6190d23609a7_8080\" template config: decoding failed due to the following error(s):\n\n'config' has invalid keys: scrape_configs"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator.(*observerHandler).startReceiver
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/observerhandler.go:222
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator.(*observerHandler).OnAdd
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/observerhandler.go:93
github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/endpointswatcher.(*EndpointsWatcher).updateAndNotifyOfEndpoints
        github.com/open-telemetry/opentelemetry-collector-contrib/extension/[email protected]/endpointswatcher/endpointswatcher.go:115
```

The fix loads the string map coming from the yaml into a `confmap.Conf`,
which is later unmarshaled properly into a `userConfigMap{}` which is
`map[string]any`.

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
#43730

<!--Describe what testing was performed and which tests were added.-->
#### Testing
1. Tuned unit-tests
2. Tested manually with the steps of the reported issue

**Deploy the helm chart using the a custom image**


(https://hub.docker.com/r/christosmarkou788/otelcontribcol-dev_rc_annotation_fix)

```yaml
mode: daemonset

image:
  repository: christosmarkou788/otelcontribcol-dev_rc_annotation_fix
  tag: "latest"
  pullPolicy: IfNotPresent

clusterRole:
  create: true
  rules:
    - apiGroups:
        - ''
      resources:
        - 'pods'
        - 'nodes'
      verbs:
        - 'get'
        - 'list'
        - 'watch'
    - apiGroups: [ "" ]
      resources: [ "nodes/proxy"]
      verbs: [ "get" ]
    - apiGroups:
        - ""
      resources:
        - nodes/stats
      verbs:
        - get
    - nonResourceURLs:
        - "/metrics"
      verbs:
        - get

config:
  extensions:
    k8s_observer:
      auth_type: serviceAccount
      node: ${env:K8S_NODE_NAME}
      observe_nodes: true
    health_check:
      endpoint: ${env:MY_POD_IP}:13133
  exporters:
    debug:
      verbosity: detailed

  receivers:
    receiver_creator/metrics:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

  service:
    extensions: [health_check, k8s_observer]
    telemetry:
      logs:
        level: INFO
    pipelines:
      metrics:
        receivers: [ receiver_creator/metrics ]
        processors: [ batch ]
        exporters: [ debug ]
```

**deploy a prometheus target**


```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: node-exporter
  name: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
      annotations:
        io.opentelemetry.discovery.metrics.9100/enabled: "true"
        io.opentelemetry.discovery.metrics.9100/scraper: prometheus
        io.opentelemetry.discovery.metrics.9100/config: |
          trim_metric_suffixes: true
          config:
            scrape_configs:
              - job_name: 'prometheus_metrics'
                scrape_interval: 10s
                metrics_path: /metrics
                static_configs:
                  - targets:
                    - '`endpoint`'
    spec:
      containers:
        - args:
            - --path.sysfs=/host/sys
            - --path.rootfs=/host/root
            - --no-collector.wifi
            - --no-collector.hwmon
            - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
            - --collector.netclass.ignored-devices=^(veth.*)$
          name: node-exporter
          image: prom/node-exporter
          ports:
            - containerPort: 9100
              protocol: TCP
          resources:
            limits:
              cpu: 250m
              memory: 180Mi
            requests:
              cpu: 102m
              memory: 180Mi
          volumeMounts:
            - mountPath: /host/sys
              mountPropagation: HostToContainer
              name: sys
              readOnly: true
            - mountPath: /host/root
              mountPropagation: HostToContainer
              name: root
              readOnly: true
      volumes:
        - hostPath:
            path: /sys
          name: sys
        - hostPath:
            path: /
          name: root
```

**Verify that discovery succeeds and the prometheus receiver starts
collecting metrics**

```console
2025-10-22T13:57:09.644Z	info	[email protected]/observerhandler.go:206	starting receiver	{"resource": {"service.instance.id": "72e07d13-c2c6-4f72-b9fe-22bd306aa3f6", "service.name": "otelcontribcol", "service.version": "0.138.0-dev"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/7616e4fa-349f-44c2-bb50-de9a98d8825b_9100", "endpoint": "10.244.0.46:9100", "endpoint_id": "k8s_observer/7616e4fa-349f-44c2-bb50-de9a98d8825b/(9100)", "config": {"config":{"scrape_configs":[{"job_name":"prometheus_metrics","metrics_path":"/metrics","scrape_interval":"10s","static_configs":[{"targets":["10.244.0.46:9100"]}]}]},"trim_metric_suffixes":true}}
2025-10-22T13:57:09.649Z	info	[email protected]/metrics_receiver.go:157	Starting discovery manager	{"resource": {"service.instance.id": "72e07d13-c2c6-4f72-b9fe-22bd306aa3f6", "service.name": "otelcontribcol", "service.version": "0.138.0-dev"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/7616e4fa-349f-44c2-bb50-de9a98d8825b_9100/receiver_creator/metrics{endpoint=\"10.244.0.46:9100\"}/k8s_observer/7616e4fa-349f-44c2-bb50-de9a98d8825b/(9100)"}
2025-10-22T13:57:09.650Z	info	targetallocator/manager.go:219	Scrape job added	{"resource": {"service.instance.id": "72e07d13-c2c6-4f72-b9fe-22bd306aa3f6", "service.name": "otelcontribcol", "service.version": "0.138.0-dev"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/7616e4fa-349f-44c2-bb50-de9a98d8825b_9100/receiver_creator/metrics{endpoint=\"10.244.0.46:9100\"}/k8s_observer/7616e4fa-349f-44c2-bb50-de9a98d8825b/(9100)", "jobName": "prometheus_metrics"}
2025-10-22T13:57:09.650Z	info	[email protected]/metrics_receiver.go:216	Starting scrape manager	{"resource": {"service.instance.id": "72e07d13-c2c6-4f72-b9fe-22bd306aa3f6", "service.name": "otelcontribcol", "service.version": "0.138.0-dev"}, "otelcol.component.id": "receiver_creator/metrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "name": "prometheus/7616e4fa-349f-44c2-bb50-de9a98d8825b_9100/receiver_creator/metrics{endpoint=\"10.244.0.46:9100\"}/k8s_observer/7616e4fa-349f-44c2-bb50-de9a98d8825b/(9100)"}
```

Signed-off-by: ChrsMark <[email protected]>

Author: ChrsMark

.chloggen/fix_autodisocvery_prometheus.yaml

@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: bug_fix
+
+# The name of the component, or a single word describing the area of concern, (e.g. receiver/filelog)
+component: receiver/receiver_creator
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Fix annotation-discovery config unmarshaling for nested configs
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [43730]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: [user]

receiver/receivercreator/discovery.go

@@ -11,6 +11,7 @@ import (
 	"strings"
 
 	"github.com/go-viper/mapstructure/v2"
+	"go.opentelemetry.io/collector/confmap"
 	"go.uber.org/zap"
 	"gopkg.in/yaml.v3"
 
@@ -187,7 +188,13 @@ func getScraperConfFromAnnotations(
 		return userConfigMap{}, nil
 	}
 	conf := userConfigMap{}
-	if err := yaml.Unmarshal([]byte(configStr), &conf); err != nil {
+
+	var rawConf map[string]any
+	if err := yaml.Unmarshal([]byte(configStr), &rawConf); err != nil {
+		return nil, err
+	}
+	cm := confmap.NewFromStringMap(rawConf)
+	if err := cm.Unmarshal(&conf); err != nil {
 		return userConfigMap{}, fmt.Errorf("could not unmarshal configuration from hint: %v", zap.Error(err))
 	}
 

receiver/receivercreator/discovery_test.go

@@ -617,7 +617,7 @@ nested_example:
 				"endpoint":            "0.0.0.0:8080",
 				"initial_delay":       "20s",
 				"read_buffer_size":    "10",
-				"nested_example":      userConfigMap{"foo": "bar"},
+				"nested_example":      map[string]any{"foo": "bar"},
 			}, defaultEndpoint: "0.0.0.0:8080",
 			scopeSuffix: "",
 		}, "simple_annotation_case_default_endpoint": {
@@ -628,7 +628,7 @@ nested_example:
 				"collection_interval": "20s",
 				"initial_delay":       "20s",
 				"read_buffer_size":    "10",
-				"nested_example":      userConfigMap{"foo": "bar"},
+				"nested_example":      map[string]any{"foo": "bar"},
 			}, defaultEndpoint: "1.1.1.1:8080",
 			scopeSuffix: "",
 		}, "simple_annotation_case_scoped": {
@@ -640,7 +640,7 @@ nested_example:
 				"endpoint":            "0.0.0.0:8080",
 				"initial_delay":       "20s",
 				"read_buffer_size":    "10",
-				"nested_example":      userConfigMap{"foo": "bar"},
+				"nested_example":      map[string]any{"foo": "bar"},
 			}, defaultEndpoint: "0.0.0.0:8080",
 			scopeSuffix: "8080",
 		}, "simple_annotation_case_with_invalid_endpoint": {