feat(prometheus): add search tools for metric names, label names and label values

[tcp13equals2]

What this changes

Adds fuzzy search of Prometheus/Mimir metric names, label names and label values, through the experimental search API (/api/v1/search/*). Today the list_prometheus_* tools match only exact names or a regex, so an assistant has to either have a strong idea of a specific metric/label/value or it has to do a broad query and then curate the full result set client side.

This fuzzy search capability allows for fuzzy string matching and relevance sort ordering. This allows for metrics, labels and values to be discovered without knowing the correct/exact spelling or without knowing the exact metric name ordering.

This is the MCP equivalent of grafana/gcx#1374.

Fixes #1274

New or changed tools

Proposal issue: #1274

Tool name	New or changed	Proposed tier
search_prometheus_metric_names	new	default-on (prometheus category)
search_prometheus_label_names	new	default-on (prometheus category)
search_prometheus_label_values	new	default-on (prometheus category)

Token cost: About +5.58% over the default tool set - this API provides a range of search options which raises this size.

Why not extend an existing tool? The search API is a different endpoint (/api/v1/search/*), with fuzzy matching, a relevance score, sorting and an NDJSON stream. If list_prometheus_* took search parameters, those tools would change behaviour, change their errors, and fail on servers without the API, which is the default on self-hosted Prometheus and Mimir.

The three tools mirror the existing list_prometheus_metric_names / _label_names / _label_values tools and gcx's three commands. A single tool with a target parameter costs less (an estimated +2.77%). It was considered, and three tools were chosen for clearer names and for a schema that enforces search and label where each one is required.

Read-only safety: Read-only. The tools send only GET requests to the search endpoints. They are discovery tools, like list_prometheus_label_values, so --disable-query does not remove them. TestProcessTools_DisableQueryRemovesOnlyQueryTools checks this.

Implementation notes

• Response handling: the NDJSON stream is read one frame at a time with a 10 MiB limit (defaultResponseLimitBytes). The API contract says clients must accept an EOF without a trailer, so in that case the tool returns the results it has read, with incomplete: true and a warning. A mid-stream error line returns an error.
• HTTP client: prometheusBackend now also holds an http.Client built on the same transport chain (auth, datasource fallback transport). It is needed because api.Client.Do in client_golang reads the whole body with no size limit.
• Input checks: inputs are validated before any request is sent: blank search terms, PromQL selectors (parsed with promql/parser), the time range, the enums and their ranges, sortBy=score without search, and sortDir without alpha. limit=0 is never sent, because Mimir reads it as unlimited and Prometheus rejects it.
• API not enabled: if the search API is not enabled, the error names the flag to turn it on and points to the matching list_prometheus_* tool. It handles Mimir's 404 feature_not_enabled and Prometheus's unavailable: search API disabled.
• Unsupported datasources: VictoriaMetrics datasources (which embed prometheusBackend) and Cloud Monitoring datasources are rejected with the same fallback hint.
• Test infrastructure: the Prometheus test image goes from 3.11.3 to 3.15.0 (sha256:efd719c9…753e, from docker buildx imagetools inspect prom/prometheus:v3.15.0), and --enable-feature=search-api is added to both starts in testdata/prometheus-entrypoint.sh. The search API starts at Prometheus 3.13.0.

How you tested it

• Unit tests: make test-unit passes. The new tests in tools/prometheus_search_test.go cover query parameter encoding and defaults, every validation rule, multi-batch streams, has_more, EOF without a trailer, the size limit, the mid-stream error line, duplicate warnings, context cancellation, both "API not enabled" hints after the datasource fallback retry, and rejection of VictoriaMetrics.
• Integration tests (docker-compose stack, Prometheus 3.15.0, through the Grafana datasource proxy): TestSearchPrometheusIntegration passes (metric names with score and metadata, label names on a selector, label values). The existing TestPrometheusTools and TestPrometheusQueries also pass on the new image. I also ran the full integration suite locally. Loki and InfluxDB pass, and so does everything else that started. MySQL and CloudWatch did not run, because other containers on my machine held their ports (3306 and 4566), so CI is the check for those.
• Cloud: TestSearchPrometheusCloud targets grafanacloud-prom on mcptests. I have not run it locally (no service-account token). I checked with gcx that the search API is enabled on that datasource: a metric-names search for alerts and a label-names search for {__name__=~".+"} both returned results.
• Lint: make lint passes (0 issues).

Checklist

• make lint and make test-unit pass locally
• Commits are signed (required to merge — see CONTRIBUTING.md)
• CLA signed (required to merge)
• README tool table updated, with RBAC permissions and scopes, if this adds a tool
• Any write operation respects --disable-write (no write operations)
• Any datasource query execution respects --disable-query (and --disable-write too, if the query is passed through unfiltered) (discovery tools, kept under --disable-query like the list_prometheus_* tools)

[cla-assistant]

All committers have signed the CLA.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[sd2k]

Love this, but to keep token usage down I think it would be good to merge this into the list_ tools with a new search parameter, and only hit the search API if the search field is set. If the search API isn't available on the server we can just return an error to that effect, which is what the search_ tools would do anyway.

If we do that, some considerations:

• Return type: the list_ tools return plain strings, while search returns objects with score and metadata. One option is to keep returning strings unless includeMetadata is set.
• Paging: list_prometheus_metric_names has a page parameter, but the search API has no cursor. Either reject page > 1 when search is set, or document that it doesn't apply.

Claude found a few other minor things:

1. tools/prometheus_search.go:35: the fuzzThreshold description says "0-100" but doesn't say that with the default jarowinkler algorithm, 0 turns fuzzy matching off and keeps only substring matches. An agent that sets 0 to get more results will get fewer.
2. tools/prometheus_search.go:~315: if the stream ends with no trailer, the tool sets HasMore=true as well as Incomplete=true. The HasMore field comment tells the agent to raise the limit, but the Incomplete comment says that won't help. If the 10 MiB cap is hit, an agent will retry with a bigger limit and make it worse. The fix is either to leave HasMore false in that case or to update the field comment.
3. tools/prometheus_search.go:31,56: the parameter names don't match the list_ tools. Search uses label where list_prometheus_label_values uses labelName, and search takes matches as PromQL strings where the list_ tools take []Selector. The schema rejects unknown fields (additionalProperties: false), so an agent that sends labelName from habit gets a schema error. (this won't matter if we merge the tools)

[tcp13equals2]

Hey Ben thanks for taking a look and the feedback!

A couple of initial comments on merging the search and list tools ...

• the search APIs are being further extended - additional search criteria and options
• the search APIs result format will continue to evolve - more decorations on the metric/label/value records
• the search API will likely support cursor based pagination soon - which is different to the fake page concept the current tool implements
• the search APIs will soon grow to also support searching over OTEL target info labels and values - whose API is a similar shape but different endpoints (see proposal: add info-metric label discovery APIs for info() autocomplete prometheus/proposals#85)
• the search APIs will likely grow to allow for searching over metric metadata and OTEL native metadata - rather then just across the raw label/values

Do you still think that given these API differences they should be abstracted behind the 3 existing list_ tools?

If the concern is about token usage, could this Prometheus Search be in it's own category and an optional thing to enable?

I don't know too much about this MCP world, so I will follow which ever direction you think is best here.

Thanks again for taking a look!

[sd2k]

@tcp13equals2 Ah that changes things, I think if things are going to evolve that much then it's worth having separate tools. You're right about the separate categories too, we could use --[enable|disable]-prometheus-search for example, disabled by default (since it's only available in certain instances) and perhaps we could enable it by default in the Cloud MCP server. A few other tips that Claude spat out when I asked about this:

• Keep it small anyway. People who turn it on still pay for every parameter. I'd still hide fuzzAlg, sortBy, sortDir and includeScore behind gcx's defaults until there's evidence agents need them, and fix the fuzzThreshold: 0 description.
• Make the parameter names match the list_ tools: labelName, and the same selector shape. An agent will see both sets of tools side by side.
• Fix the HasMore/Incomplete contradiction from the review.
• Make the docs say when to use it. The tool descriptions should tell an agent to prefer search when it's available. Otherwise agents that have both sets will keep defaulting to list_.
• Plan to make it a default later. Once the API is on by default in Mimir and Grafana Cloud, it's worth revisiting turning the category on by default, at least
  for the cloud server. Otherwise it stays a feature hardly anyone uses.

I think we can get this in though.

In other news, the v2 branch changed from the mark3labs/mcp-go SDK to the official Go SDK so there are quite a few conflicts, sorry about that 🙈

[tcp13equals2]

Thanks @sd2k ... I'll get some further updates to these search APIs into Prometheus / Mimir, do another round of gcx updates and then update this PR to address your comments / leverage the latest API updates.

Thank you for your comments and direction!