Skip to content

Conversation

@qinheping
Copy link
Contributor

Description of changes:

The current benchcomp compares the two versions (pre/post-PR commits) of Kani on the corresponding two versions of benchmarks. It make no sense to compare their performance if the commit modified the performance benchmarks.

In this PR, we copy the post-PR benchmarks to the pre-PR folder to make sure benchcomp always run on the same set of benchmarks.

Here is the CI run result with this commit
https://github.com/qinheping/kani/actions/runs/4879281724/jobs/8705705561?pr=3
comparing to the current workflow
https://github.com/model-checking/kani/actions/runs/4866466752/jobs/8678021607?pr=2422

Resolved issues:

Resolves #2424

Related RFC:

Optional #ISSUE-NUMBER.

Call-outs:

Testing:

  • How is this change tested?

  • Is this a refactor change?

Checklist

  • Each commit message has a non-empty body, explaining why the change was made
  • Methods or procedures are documented
  • Regression or unit tests are included, or existing tests cover the modified code
  • My PR is restricted to a single feature or bugfix

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

@qinheping qinheping requested a review from a team as a code owner May 4, 2023 15:51
Copy link
Contributor

@celinval celinval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this.

@karkhaz do you have any plan to address this issue as part of the tool or the configuration script?

@qinheping qinheping enabled auto-merge (squash) May 4, 2023 20:55
@qinheping qinheping merged commit 8e6267d into model-checking:main May 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adding a performance regression breaks perf-benchcomp CI workflow

3 participants