elastic-agent: update monitor fields to include new TBS metrics

[rubvs]

Proposed commit message

Add back apm-server monitoring metrics removed in #12793, plus new TBS metrics outlined in elastic/apm-server#15533

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Verify mappings are present in the metrics-elastic_agent.apm_server* datastream.

How to test this PR locally

Step 1: Ensure Elasticsearch & Kibana is running

1. Spin up a stack in QA: https://console.qa.cld.elstc.co/

> cd integrations/packages/elastic_agent

> export ELASTIC_PACKAGE_KIBANA_HOST=https://alice-863450.kb.us-central1.gcp.qa.cld.elstc.co
> export ELASTIC_PACKAGE_ELASTICSEARCH_HOST=https://alice-863450.es.us-central1.gcp.qa.cld.elstc.co
> export ELASTIC_PACKAGE_ELASTICSEARCH_USERNAME="elastic"
> export ELASTIC_PACKAGE_ELASTICSEARCH_PASSWORD="xxx"

> elastic-package stack up -v -d --provider environment

2. Check stack health

> elastic-package stack status

Status of Elastic stack services:
╭───────────────┬─────────┬───────────────────╮
│    SERVICE    │ VERSION │      STATUS       │
├───────────────┼─────────┼───────────────────┤
│ elasticsearch │ 9.2.1   │ healthy           │
│ kibana        │ 9.2.1   │ healthy           │
│ fleet-server  │ unknown │ healthy           │
│ elastic-agent │ 9.2.1   │ running (healthy) │
╰───────────────┴─────────┴───────────────────╯

Step 2: Apply Integration Changes

• Install the elastic_agent integration with the updated mappings in this PR

> cd integrations/packages/elastic_agent

> elastic-package build
> elastic-package stack up -v -d --services package-registry
> elastic-package install -v

Step 3: Enroll Elastic Agent in Fleet

1. Go to the QA Kibana UI
2. Data management -> Integrations -> Installed integrations -> Elastic APM -> Add Elastic APM
3. Add Elastic Agent to your hosts -> Add agent -> Go to Step 4

Step 4: Install EA Locally in a Container

1. Create docker container to run EA in.

> docker run --rm -it -p 8220:8220 -p 8200:8200 buildpack-deps:24.04 /bin/bash

2. From within the container, update and add neovim to edit apm-server.yml later on.

> apt update -y && apt install -y neovim

4. Install elastic-agent with commands from Kibana UI listed in Step 3.

> curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.2.1-linux-arm64.tar.gz 
> tar xzvf elastic-agent-9.2.1-linux-arm64.tar.gz
> cd elastic-agent-9.2.1-linux-arm64

> ./elastic-agent install --url=<REDACTED> --enrollment-token=<REDACTED> --install-servers

5. Check the Elastic Agent health

> elastic-agent status

┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   └─ status: (HEALTHY) Running

Step 5: Verify Data

Verify mappings in the metrics-elastic_agent.apm_server* datastream from Dev Tools.

1. Get the index name by running the cmd below and copying the data_streams.indices.index_name field.

GET _data_stream/metrics-elastic_agent.apm_server*

2. Contains the proper mappings due to the changes made in this PR.

GET .ds-metrics-elastic_agent.apm_server-default-2025.11.19-000001/_mapping?filter_path=**.apm_server.**.storage

{
  ".ds-metrics-elastic_agent.apm_server-default-2025.11.19-000002": {
    "mappings": {
      "properties": {
        "beat": {
          "properties": {
            "stats": {
              "properties": {
                "apm_server": {
                  "properties": {
                    "sampling": {
                      "properties": {
                        "tail": {
                          "properties": {
                            "storage": {
                              "properties": {
                                "disk_total": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                },
                                "disk_usage_threshold": {
                                  "type": "long",
                                  "time_series_metric": "gauge"
                                },
                                "disk_used": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                },
                                "lsm_size": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                },
                                "storage_limit": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                },
                                "value_log_size": {
                                  "type": "long",
                                  "meta": {
                                    "unit": "byte"
                                  },
                                  "time_series_metric": "gauge"
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Related issues

• Relates monitoring: expose storage limit and disk metrics for TBS monitoring apm-server#15533

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 8a44d66

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

@rubvs I followed the testing procedure and the new mappings are not there. I even tried to roll over the index by running:

POST metrics-elastic_agent.apm_server-default/_rollover

However it did not work.

One thing I noticed is that the PR is missing the version update in manifest.yml and changelog in changelog.yml.

When I checked those files they show version 2.6.8 but on my Kibana I can see the Elastic Agent integration in version 2.6.9.

Could you merge main into your PR and add the changelog?

[ericywl]

Hi @belimawr, Ruben has left the company so I will try to follow up with this 👍