Prometheus send failed, despite not being enabled

[lukasdietrich]

Description

Hello,

k8up tries to send stats to prometheus, despite not being enabled or otherwise configured.

This does not impact the functionality, it is just an error in the logs.

Additional Context

I have installed k8up using helm with metrics disabled (metrics.serviceMonitor.enabled, metrics.prometheusRule.enabled and metrics.grafanaDashboard.enabled set to false).

Then I have set up a k8up.io/v1/Schedule without a promURL.
The schedule triggers successful backup jobs, but logs an error, that it could not send stats to prometheus.

The job has a PROM_URL set as an environment variable:

spec:
  containers:
    - args:
      - -varDir
      - /k8up
      command:
      - /usr/local/bin/k8up
      - restic
      env:
      - name: PROM_URL
        value: http://127.0.0.1/
      ...

Logs

2026-03-26T14:02:39Z    INFO    k8up.restic.statsHandler.promStats    sending prometheus stats    {"url": "http://127.0.0.1/"}
2026-03-26T14:02:39Z    ERROR    k8up.restic.restic    prometheus send failed    {"error": "Post \"http://127.0.0.1/metrics/job/restic_backup/instance/vaultwarden-app/cluster/default\": dial
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).sendBackupStats
    /home/runner/work/k8up/k8up/restic/cli/backup.go:126
github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out
    /home/runner/work/k8up/k8up/restic/logging/logging.go:162
github.com/k8up-io/k8up/v2/restic/logging.writer.Write
    /home/runner/work/k8up/k8up/restic/logging/logging.go:103
io.copyBuffer
    /opt/hostedtoolcache/go/1.26.1/x64/src/io/io.go:431
io.Copy
    /opt/hostedtoolcache/go/1.26.1/x64/src/io/io.go:388
os.genericWriteTo
    /opt/hostedtoolcache/go/1.26.1/x64/src/os/file.go:295
os.(*File).WriteTo
    /opt/hostedtoolcache/go/1.26.1/x64/src/os/file.go:273
io.copyBuffer
    /opt/hostedtoolcache/go/1.26.1/x64/src/io/io.go:411
io.Copy
    /opt/hostedtoolcache/go/1.26.1/x64/src/io/io.go:388
os/exec.(*Cmd).writerDescriptor.func1
    /opt/hostedtoolcache/go/1.26.1/x64/src/os/exec/exec.go:602
os/exec.(*Cmd).Start.func2
    /opt/hostedtoolcache/go/1.26.1/x64/src/os/exec/exec.go:757
2026-03-26T14:02:39Z    INFO    k8up.restic.restic.backup    backup finished, sending snapshot list
2026-03-26T14:02:39Z    INFO    k8up.restic.restic.snapshots    getting list of snapshots
2026-03-26T14:02:39Z    INFO    k8up.restic.restic.snapshots.command    restic command    {"path": "/usr/local/bin/restic", "args": ["snapshots", "--json"]}
2026-03-26T14:02:39Z    INFO    k8up.restic.restic.snapshots.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}

Expected Behavior

The job should not have a (default?) PROM_URL set.

Steps To Reproduce

No response

Version of K8up

v2.14.0 (Chart Version = 4.8.7)

Version of Kubernetes

v1.35.2+k3s1

Distribution of Kubernetes

k3s

[reinout]

I'm seeing the same behaviour.
Note that I haven't actively disabled the metrics stuff, I just have not enabled it.
My expectation was that it won't try to send metrics in that case.

[lukasdietrich]

After reading more of the documentation, I have found a workaround.
You can set an empty global prometheus url on the operator, which prevents the jobs from defaulting to 127.0.0.1.

The relevant variable is BACKUP_PROMURL (see https://docs.k8up.io/k8up/2.15/references/operator-config-reference.html).

Setting environment variables is already supported by the helm chart:

k8up/charts/k8up/values.yaml

Line 32 in d01fae8

envVars: []

[reinout]

• You're right, the 127.0.0.1 is documented.
• I just set the environment variable you mentioned to an empty string and, yes, there are no more errors.
• How useful is a 127.0.0.1 default? If people set up prometheus, it probably runs on some.external.provider or central.internal.installation, not on 127.0.0.1, I guess?

[Nagasirikancha]

/assign