Skip to content

Conversation

@kukushechkin
Copy link
Contributor

@kukushechkin kukushechkin commented Nov 11, 2025

Run benchmarks on macOS in addition to Linux.

Motivation:

  1. macOS and Linux benchmark yield different results and we would like to rack both of them.
  2. default Linux GitHub workers do not provide access to PMU, which is necessary for instructions metric benchmarking

Modifications:

  • benchmarks.yml workflow support running benchmarks on self-hosted macOS runners

Result:

Benchmarks are running on both linux and macOS runners. An example of an opt-in PR running just one Xcode version: https://github.com/apple/swift-nio/actions/runs/19468002318/job/55707591945.

An example of an opt-in PR and thresholds expectations check failed:
https://github.com/apple/swift-nio/actions/runs/19468236909/job/55708359111

I've been running swift-nio benchmarks on macOS runners and it is much less stable than Linux for allocations and context switches. I suggest we make this an opt-in feature and take in only for instructions count benchmarks, for example upcoming to the swift-log. This also makes it less stressful for macOS runners pool.

@FranzBusch
Copy link
Member

I know this is still a draft but I think we need to consider resource capacity here. The reason we haven't been running benchmarks on macOS is due to the limited number of available macOS runners. Let's explore if we can get custom Linux runners that provide access to the PMU.

@kukushechkin kukushechkin added the semver/none No version bump required. label Nov 11, 2025
@kukushechkin kukushechkin force-pushed the run-benchmarks-in-CI-on-macOS branch from 640f3bb to fbbe95c Compare November 17, 2025 14:22
@kukushechkin kukushechkin marked this pull request as ready for review November 17, 2025 15:15
@kukushechkin kukushechkin requested a review from rnro November 17, 2025 15:15
Copy link
Member

@FranzBusch FranzBusch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks sensible to me since it allows user's to run benchmarks on macOS but those are disabled by default. Would love for @rnro to also take a look

@kukushechkin kukushechkin requested a review from rnro November 18, 2025 13:17
Copy link
Contributor

@rnro rnro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@kukushechkin kukushechkin merged commit 6dae988 into apple:main Nov 18, 2025
106 of 107 checks passed
@kukushechkin kukushechkin deleted the run-benchmarks-in-CI-on-macOS branch November 18, 2025 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver/none No version bump required.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants