EDOT Daemon Collector: Enrich apm pipelines with k8s information

[osullivandonal]

Describe the enhancement:

We would like to enrich the apm pipelines (logs, metrics, and traces) with k8sattributes information. Currently in the otel-demo we do not have k8s information available per pod.

This update would add the k8sattributes infromation the apm pipelines to provide k8s information on the pod level, not just on the node level.

Describe a specific use case for the enhancement or feature:

This will populate Kibana with k8s information, for example for services that do not auto instrument.

Here is some examples of before and after from doing this:

Before

After

NB this add k8sattributes to the apm pipelines for the daemon collector:

Also in the Discover dashboard:

Before

After

Work involved

Update the EDOT Daemon collector k8sattributes, and apm pipelines:

            extract:
              metadata:
                - "k8s.namespace.name"
                - "k8s.deployment.name"
                - "k8s.replicaset.name"
                - "k8s.statefulset.name"
                - "k8s.daemonset.name"
                - "k8s.cronjob.name"
                - "k8s.job.name"
                - "k8s.node.name"
                - "k8s.pod.name"
                - "k8s.pod.ip"
                - "k8s.pod.uid"
                - "k8s.pod.start_time"
                - "container.id"              # <-- add
                - "service.name"
                - "service.version"
                - "service.instance.id"
        service:
          pipelines:
            traces/apm:
              receivers:
                - otlp
              processors:
                - k8sattributes           # <-- add
                - batch
                - resource/hostname
              exporters:
                - otlp/gateway
            logs/apm:
              receivers:
                - otlp
              processors:
                - k8sattributes           # <-- add
                - batch
                - resource/hostname
              exporters:
                - otlp/gateway
            metrics/otel-apm:
              receivers:
                - otlp
              processors:
                - k8sattributes           # <-- add
                - batch/metrics
                - resource/hostname
              exporters:
                - otlp/gateway

cc. @sboomsma @girodav
What is the definition of done?

The APM pipelines contain k8sattributes information.

[osullivandonal]

This also allows us to investigate the pod logs and metrics:

Before

After (Wtith k8s enrichment)