PMM-14737 QAN dev env.

[JiriCtvrtka]

PMM-14737

FB: Percona-Lab/pmm-submodules#4201

We dont need update timestamps, because if data get deleted we can easily recover them by one command. Also it is good to stick with same data and period range when performing benchmark on different PMM versions.

Repository to store PMM Demo backups: https://github.com/Percona-Lab/pmm-demo-dump

Usage:

make download-clickhouse-backup
make restore-clickhouse-backup
make bench

Variables:

BACKUP_NAME ?= 20260120
BACKUP_LAST_MIGRATION ?= 21
PMM_DEMO_BENCH_PERIOD_FROM ?= 2025-12-27T00:00:00+01:00
PMM_DEMO_BENCH_PERIOD_TO ?= 2026-01-21T23:59:59+01:00

Description
1. Download the PMM Demo backup
make download-clickhouse-backup

The default backup is 2026-01-20.

All available backups can be found here: https://github.com/Percona-Lab/pmm-demo-dump/releases/tag/pmm-demo

To download a different backup, set the Makefile variable:
BACKUP_NAME=<backup_name>

2. Restore the PMM Demo backup
After downloading, the backup will be available in "dev/clickhouse-backups/"

Example: dev/clickhouse-backups/20260120

Restore the backup by running:
make restore-clickhouse-backup

You must specify the back name and latest migration applied in the backup.
For the default backup (20260120), this is migration 21 (default).

To restore a different backup, set the Makefile variable:
BACKUP_NAME=<backup_name>
BACKUP_NAME=<backup_last_migration>

For other backups, the latest applied migration is documented in the release description:
https://github.com/Percona-Lab/pmm-demo-dump/releases/tag/pmm-demo

3. Run benchmarks
To benchmark all main endpoints:
make bench

To benchmark specific endpoints:

make bench-filters
make bench-report
make bench-metrics
make bench-example

Each benchmark runs 10 iterations and reports AVG, MIN, and MAX timings.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 45.16%. Comparing base (435cc48) to head (555f1f1).
⚠️ Report is 1 commits behind head on v3.

Additional details and impacted files

@@            Coverage Diff             @@
##               v3    #4955      +/-   ##
==========================================
- Coverage   45.83%   45.16%   -0.67%     
==========================================
  Files         367      367              
  Lines       38531    38531              
==========================================
- Hits        17659    17402     -257     
- Misses      19172    19458     +286     
+ Partials     1700     1671      -29     

Flag	Coverage Δ
admin	17.82% <ø> (ø)
agent	50.72% <ø> (-2.68%)	⬇️
managed	46.45% <ø> (ø)
vmproxy	72.09% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[JiriCtvrtka]

Real changes in dev config compared to default in PMM Server. Other lines are copy out from there.

[maxkondr]

I think it is worth to explicitly describe that these MAKE targets shall be running outside of docker container. Because right now we have a mix: some shall be run on host, the other in pmm-server docker container (like make run-managed).

[maxkondr]

I would add the possibility to override credentials via ENV

[JiriCtvrtka]

Does it make sense if this is only for development purposes and we have static credentials in ClickHouse? It is not included in the image, and customers cannot use it anyway.

[maxkondr]

this is really for development purposes but we can't guarantee that this script will be used with our default CH deployment only. What if developers spins up his separate deployment 9another CH version, cluster-based once, etc) and needs to run benchmarks

[JiriCtvrtka]

Option to pass user and password via variables is added.

[JiriCtvrtka]

With $(info ) it always printed it to output in my case.

[maxkondr]

this is not quite correct measurements because:

1. it measures request preparation in benchmarkRequest
2. it measures HTTP client initialization
3. it measures I/O in b.Logf() call

I would suggest making the following:

1. exclude request preparation (setup) from measurements:

func benchmarkRequest(b *testing.B, ...) time.Duration {
  <init client>
  <init request>
  
  start := time.Now()  
  b.ResetTimer() // <--- start measurement here
  resp, err := client.Do(req)
  b.StopTimer() // <----- in order to exclude the rest of the function from measurement
  duration := time.Since(start)
  if err != nil {
	b.Fatalf("Request failed: %v", err)
  }
  defer resp.Body.Close()
  if resp.StatusCode != http.StatusOK {
	b.Fatalf("Unexpected status code: %d", resp.StatusCode)
  }

  return duration
}

One more point - use testing.B.Loop() - https://go.dev/blog/testing-b-loop

func BenchmarkGetFilters(b *testing.B) {
	url := baseURL + "qan/metrics:getFilters"
	payload := `{
		"labels": [],
		"main_metric_name": "load"}`
    f := prepareRequest(...)

    for b.Loop() {
  	   benchmarkRequest(b, url, payload)
    }
}

in order to print an additional you may use https://pkg.go.dev/testing#B.ReportMetric

[JiriCtvrtka]

1. Loop is applied.
2. ReportMetric is used.
3. Suggestion above

func benchmarkRequest(b *testing.B, ...) time.Duration {
  <init client>
  <init request>
  
  start := time.Now()  
  b.ResetTimer() // <--- start measurement here
  resp, err := client.Do(req)
  b.StopTimer() // <----- in order to exclude the rest of the function from measurement
  duration := time.Since(start)
  if err != nil {
	b.Fatalf("Request failed: %v", err)
  }
  defer resp.Body.Close()
  if resp.StatusCode != http.StatusOK {
	b.Fatalf("Unexpected status code: %d", resp.StatusCode)
  }

  return duration
}

results in the following error:
"benchmark.go:412: B.Loop called with timer stopped"
The timer is stopped and execution continues with the next iteration, so I added a Start call before End to fix this. However, resetting and stopping the timer will slightly affect the measured time, which is the main indicator we care about in this benchmark.