fix: allow legacy name validation scheme for prometheus metrics

.spelling

@@ -129,6 +129,7 @@ Traefik
 Triaging
 TripAdvisor
 UI
+UTF-8
 VSCode
 Valasek
 Webhooks

docs/environment-variables.md

@@ -44,6 +44,7 @@ This document outlines environment variables that can be used to customize behav
 | `OFFLOAD_NODE_STATUS_TTL`                | `time.Duration`     | `5m`                                                                                        | The TTL to delete the offloaded node status. Currently only used for testing.                                                                                                                                                                                            |
 | `OPERATION_DURATION_METRIC_BUCKET_COUNT` | `int`               | `6`                                                                                         | The number of buckets to collect the metric for the operation duration.                                                                                                                                                                                                  |
 | `POD_NAMES`                              | `string`            | `v2`                                                                                        | Whether to have pod names contain the template name (v2) or be the node id (v1) - should be set the same for Argo Server.                                                                                                                                                |
+| `PROMETHEUS_LEGACY_NAME_VALIDATION_SCHEME`                              | `bool`            | `false`                                                                                        | Makes Prometheus emit metrics with only underscores as delimiters by setting `NameValidationScheme` to `LegacyValidation`.   |
 | `RECENTLY_STARTED_POD_DURATION`          | `time.Duration`     | `10s`                                                                                       | The duration of a pod before the pod is considered to be recently started.                                                                                                                                                                                               |
 | `RECENTLY_DELETED_POD_DURATION`          | `time.Duration`     | `2m`                                                                                       | The duration of a pod before the pod is considered to be recently deleted.                                                                                                                                                                                                |
 | `TASK_RESULT_TIMEOUT_DURATION`           | `time.Duration`     | `10m`                                                                                       | The duration of time before a node is marked completed but the `taskresult` has not arrived yet, this is a more general and more conservative version of `RECENTLY_DELETED_POD_DURATION` that is used when a `taskresult` hasn't arrived but the pod is still around.    |

docs/metrics.md

@@ -124,6 +124,9 @@ Assuming you only have one controller replica, you can port-forward with:
 kubectl -n argo port-forward deploy/workflow-controller 9090:9090
 ```
 
+!!! Note "UTF-8 in Prometheus metrics"
+    Version `v3.7` upgraded the `github.com/prometheus/client_golang` library, changing the `NameValidationScheme` to `UTF8Validation`. This allows metric names to retain their original delimiters (e.g., .), instead of replacing them with underscores. To maintain the legacy behavior, you can set the environment variable `PROMETHEUS_LEGACY_NAME_VALIDATION_SCHEME`. For more details, refer to the official [Prometheus documentation](https://prometheus.io/docs/guides/utf8/).
+
 ### Common
 
 You can adjust various elements of the metrics configuration by changing values in the [Workflow Controller Config Map](workflow-controller-configmap.md).

util/telemetry/exporter_prometheus.go

@@ -5,11 +5,13 @@ import (
 	"crypto/tls"
 	"fmt"
 	"net/http"
+	"os"
 	"time"
 
 	"github.com/argoproj/argo-workflows/v3/util/logging"
 
 	promgo "github.com/prometheus/client_golang/prometheus"
+	prommodel "github.com/prometheus/common/model"
 	"go.opentelemetry.io/otel/exporters/prometheus"
 
 	"github.com/prometheus/client_golang/prometheus/promhttp"
@@ -19,6 +21,14 @@ import (
 	tlsutils "github.com/argoproj/argo-workflows/v3/util/tls"
 )
 
+// by default prometheus has NameValidationScheme set to UTF8Validation which allows metrics names to keep original delimiters (e.g. .),
+// rather than replacing with underscores. This flag allows the user to revert to the legacy behavior.
+func init() {
+	if _, legacyValidationScheme := os.LookupEnv("PROMETHEUS_LEGACY_NAME_VALIDATION_SCHEME"); legacyValidationScheme {
+		prommodel.NameValidationScheme = prommodel.LegacyValidation //nolint:staticcheck
+	}
+}
+
 const (
 	DefaultPrometheusServerPort = 9090
 	DefaultPrometheusServerPath = "/metrics"