-
Notifications
You must be signed in to change notification settings - Fork 719
Run benchmarks in CI on macOS in addition to linux #3444
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run benchmarks in CI on macOS in addition to linux #3444
Conversation
|
I know this is still a draft but I think we need to consider resource capacity here. The reason we haven't been running benchmarks on macOS is due to the limited number of available macOS runners. Let's explore if we can get custom Linux runners that provide access to the PMU. |
640f3bb to
fbbe95c
Compare
FranzBusch
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall this looks sensible to me since it allows user's to run benchmarks on macOS but those are disabled by default. Would love for @rnro to also take a look
rnro
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, thanks!
Run benchmarks on macOS in addition to Linux.
Motivation:
Modifications:
benchmarks.ymlworkflow support running benchmarks on self-hosted macOS runnersResult:
Benchmarks are running on both linux and macOS runners. An example of an opt-in PR running just one Xcode version: https://github.com/apple/swift-nio/actions/runs/19468002318/job/55707591945.
An example of an opt-in PR and thresholds expectations check failed:
https://github.com/apple/swift-nio/actions/runs/19468236909/job/55708359111
I've been running
swift-niobenchmarks on macOS runners and it is much less stable than Linux for allocations and context switches. I suggest we make this an opt-in feature and take in only for instructions count benchmarks, for example upcoming to theswift-log. This also makes it less stressful for macOS runners pool.