[Elastic Agent] Updates to Elastic Agent metrics dashboards

[strawgate]

Fixes #12488

Replaces closed PR: #12524

Benefits from:

• Include full queue metrics in the monitoring index beats#42439
• Write latency statistics go under a histogram key beats#42514
• Wire in Kafka read and write metrics beats#42537
• Add latency metrics for logstash async output beats#42565

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Related issues

Screenshots

Overview

Concerning Agents

Agent Metrics

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e33d59a

[elastic-sonarqube]

Quality Gate failed

Failed conditions
75.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[leehinman]

One question on the interval, one title with "copy" in it.

But overall looks ok to me.

[leehinman]

I'm not sure if this is actually visible to the end user or not (didn't see it in screenshots), but we probably don't want copy in the title.

[leehinman]

I think for anything that is a counter rate we want to keep the interval at "m" or "30s", we don't get the data very frequently and you don't want your interval to not have records in it to perform the counter rate, that leads to very odd looking graphs. Unless something in Kibana has fixed this.

[belimawr]

@strawgate do you have any tip/trick to add data to populate test the dashboard?

The screenshots look good, but I'm not sure I can easily generate data to populate all dashboards.

[strawgate]

@strawgate do you have any tip/trick to add data to populate test the dashboard?

The screenshots look good, but I'm not sure I can easily generate data to populate all dashboards.

I just run a cgroup limited test agent in docker pointed at the cluster with the dashboards.

docker run --env FLEET_ENROLL=1 --memory 1gb --cpus 0.5  --env FLEET_URL=https://mycluster.fleet.us-central1.gcp.cloud.es.io:443  --env FLEET_ENROLLMENT_TOKEN=mykey  --rm docker.elastic.co/elastic-agent/elastic-agent:8.11.2

[leehinman]

The screenshots look good, but I'm not sure I can easily generate data to populate all dashboards.

For the failed & rejects to Elasticsearch this is why I built https://github.com/elastic/mock-es, so you can leave the monitoring data going to a "real" elasticsearch, but have the integrations send to mock-es configured to have some errors. You do have to set the clusteruuid option in mock-es.

[belimawr]

LGTM as long as Lee's comments are addressed.

[belimawr]

Was automerge enabled? I totally missed that 🤦‍♂️

[elastic-vault-github-plugin-prod]

Package elastic_agent - 2.1.0 containing this change is available at https://epr.elastic.co/package/elastic_agent/2.1.0/