rapids-cmake can generate pinned versions file by robertmaynard · Pull Request #530 · rapidsai/rapids-cmake

robertmaynard · 2024-01-30T21:13:14Z

Description

Add rapids_cpm_generate_pinned_versions to support reproducible builds

rapids-cmake can now generate a versions.json that contains the exact git SHA1 used by depdendencies which can be re-used
to construct reproducible builds.

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.
The cmake-format.json is up to date with these changes.
I have added new files under rapids-cmake/
- I have added include guards (include_guard(GLOBAL))
- I have added the associated docs/ rst file and update the api.rst

… they are testing

rapids-cmake can now generate a `versions.json` that contains the exact git SHA1 used by depdendencies which can be re-used to construct reproducible builds.

rapids-cmake/cpm/detail/gpv_root_dir_hook.cmake

rapids-cmake/cpm/detail/gpv_write_file.cmake

rapids-cmake/cpm/init.cmake

testing/cpm/cpm_generate_pins-format-patches/override.json

testing/cpm/cpm_generate_pins-format-patches/patches/hide_kernels.patch

testing/export/write_language-multiple-nested-enables/CMakeLists.txt

Co-authored-by: Bradley Dice <[email protected]>

…ts.txt Co-authored-by: Bradley Dice <[email protected]>

rapids-cmake/cpm/detail/pinning_write_file.cmake

rapids-cmake/cpm/generate_pinned_versions.cmake

Co-authored-by: Bradley Dice <[email protected]>

bdice

Yay! Thanks so much for your hard work and many iterations. This has improved a lot from the initial draft. 🙇‍♂️

vyasr

Do we want to make rapids-cpm-init respect an environment variable/CMake option so that users can do something like cmake ... -DRAPIDS_CMAKE_GENERATE_PINNED_VERSIONS (name TBD)? Or do we expect that consumers of this feature will modify their calls to rapids_cpm_init?

I haven't reviewed tests yet, will do that in a second pass after we get through this first round of questions.

rapids-cmake/cpm/generate_pinned_versions.cmake

rapids-cmake/cpm/detail/pinning_root_dir_hook.cmake

rapids-cmake/cpm/detail/pinning_write_file.cmake

vyasr

My question on the overall review (not tied to any line) from the last review remains outstanding, but I don't think it's super important. The other remaining items are pretty small and we should be good to finish this up in one more round.

rapids-cmake/cpm/detail/pinning_write_file.cmake

testing/utils/fill_cache/CMakeLists.txt

testing/other/CMakeLists.txt

testing/cpm/cpm_generate_pins-nested/verify/CMakeLists.txt

vyasr · 2024-03-01T00:56:25Z

testing/cpm/cpm_generate_pins-nested/verify/CMakeLists.txt

+  # Everything should have shallow marked as false
+  # so that clones by SHA1 work
+  if(${proj}_shallow)
+    message(FATAL_ERROR "${proj}_shallow} is expected to be false, but got ${${proj}_shallow}")
+  endif()


Out of scope for this PR, but given the amount of duplication required to write CMake tests it might be good for us to centralize helpers that get used in multiple tests.

testing/cpm/cpm_generate_pins-nested/verify/CMakeLists.txt

testing/cpm/cpm_generate_pins-format-patches/CMakeLists.txt

robertmaynard · 2024-03-05T13:48:51Z

/merge

Modifies pinning tests from #530: * to only test projects that were downloaded by CPM (e.g. ignoring the `fmt` that might already exist in the build environment) * to echo out `pinned_versions.json` and `versions.json` in logs from failed tests, to make debugging faster ## Notes for Reviewers #592 proposes some testing changes that aren't specific to the goals of that PR. Since that PR might be stuck for a bit (rapidsai/build-planning#56 (comment)), this proposes pulling those out into a separate PR: * so that other changes in this project benefit from them * to shrink the diff of #592 and therefore the risk of merge conflicts ### How I tested this Pushed a commit with the new test error message content changes but keeping `fmt` in the failing tests, to confirm that the expected tests failed. <details><summary>got the expected outputs (click me)</summary> ```text The following tests FAILED: 698 - cpm_generate_pins-nested-makefile (Failed) 700 - cpm_generate_pins-nested-ninja (Failed) 702 - cpm_generate_pins-nested-ninja_multi-config (Failed) 722 - cpm_generate_pins-simple-makefile (Failed) 724 - cpm_generate_pins-simple-ninja (Failed) 726 - cpm_generate_pins-simple-ninja_multi-config (Failed) ``` And they failed in the expected way more informative logs! ```text CMake Error at CMakeLists.txt:51 (message): pinned fmt tag (10.2.1) should differ compared to baseline 10.2.1 pinned_versions.json: { "always_download" : true, "git_shallow" : false, "git_tag" : "${version}", "git_url" : "https://github.com/fmtlib/fmt.git", "version" : "10.2.1" } versions.json: { "git_tag" : "${version}", "git_url" : "https://github.com/fmtlib/fmt.git", "version" : "10.2.1" } ``` ([build link](https://github.com/rapidsai/rapids-cmake/actions/runs/8837613213/job/24267079292?pr=592)) </details> # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Robert Maynard (https://github.com/robertmaynard) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #599

robertmaynard requested a review from a team as a code owner January 30, 2024 21:13

robertmaynard changed the title ~~Fea/generate pinned manifest file~~ rapids-cmake can generate pinned versions file Jan 30, 2024

robertmaynard added 2 commits January 31, 2024 15:54

All rapids-cmake tests will properly hook into using the rapids-cmake…

a24d84b

… they are testing

Add rapids_cpm_generate_pinned_versions to support reproducible builds

cbba2a0

rapids-cmake can now generate a `versions.json` that contains the exact git SHA1 used by depdendencies which can be re-used to construct reproducible builds.

robertmaynard force-pushed the fea/generate_pinned_manifest_file branch from ec84328 to cbba2a0 Compare January 31, 2024 20:54

robertmaynard added feature request New feature or request non-breaking Introduces a non-breaking change labels Jan 31, 2024

robertmaynard added 2 commits January 31, 2024 16:41

Ensure we can apply git format-patches always

cfe884b

add debug code to failing test

0d0111e

robertmaynard force-pushed the fea/generate_pinned_manifest_file branch from bfd2ffc to 0d0111e Compare January 31, 2024 21:41

robertmaynard added 5 commits January 31, 2024 17:03

Fix logic found by CI

f93edc8

Fix docs issues

86f4c2e

Fix docs issues

cb613cf

Fix docs issues

2a97e9d

Refactor internal logic to remove unneeded hooks

51cf575

bdice reviewed Feb 22, 2024

View reviewed changes

robertmaynard and others added 7 commits February 22, 2024 11:48

Apply suggestions from code review

6d65041

Co-authored-by: Bradley Dice <[email protected]>

Correct git captilization

cf29f15

Remove unneeded files from cpm pin format-patches test

90b4a98

Update testing/export/write_language-multiple-nested-enables/CMakeLis…

cdea83c

…ts.txt Co-authored-by: Bradley Dice <[email protected]>

Cleaned up how we handle trailing commas

e351cdb

Correct style issues found by CI

e15e952

Refactor gpv name to pinning

5ce2caf

bdice reviewed Feb 23, 2024

View reviewed changes

Remove unneeded whitespace from json

9101346

robertmaynard mentioned this pull request Feb 26, 2024

Remove unneeded whitespace from json #544

Merged

2 tasks

Refactor rapids_cpm_pinning_create_and_set_member

1fc2b4b

robertmaynard requested a review from bdice February 26, 2024 16:33

bdice reviewed Feb 26, 2024

View reviewed changes

Apply suggestions from code review

dd87f9d

Co-authored-by: Bradley Dice <[email protected]>

bdice approved these changes Feb 27, 2024

View reviewed changes

robertmaynard and others added 2 commits February 27, 2024 10:39

Merge branch 'branch-24.04' into fea/generate_pinned_manifest_file

af13b6c

Apply pre-commit hooks for pretty-format-json.

22a849c

vyasr requested changes Feb 28, 2024

View reviewed changes

robertmaynard added 2 commits February 28, 2024 15:52

Correct issues found by code review

a6b4cb9

Leverage CMake for pretty enough printing

7cb161f

vyasr requested changes Mar 1, 2024

View reviewed changes

robertmaynard added 7 commits March 1, 2024 09:32

More corrections found in code review

e0a9c47

Document functions in pinning_write_file.cmake

dc0e01f

Update tests to pass in the projects to verify

6a6a6ef

Update tests to pass in the projects to verify

df1567a

Remove unneeded variable expansion from tests

5a5081b

Correctly handle CPM calls that resolve to find_package

6ad1956

Correct style issues found by CI

49b4035

bdice mentioned this pull request Mar 4, 2024

[BUG] The build can be broken due to dependency changes of cudf NVIDIA/spark-rapids-jni#1666

Closed

robertmaynard requested a review from vyasr March 4, 2024 13:15

robertmaynard added 5 commits March 4, 2024 15:04

Add messaging around non source packages

7542f30

Correct style issues found by CI

97a8b45

Correct style issues found by CI

80ca70c

reduce verbosity of rapids_cpm_generate_pinned_versions messasges

75e0db8

Correct style issues found by CI

f341f37

vyasr approved these changes Mar 4, 2024

View reviewed changes

robertmaynard mentioned this pull request Mar 5, 2024

[FEA] rapids-cmake pinning override allow generation via CMake variable #552

Closed

rapids-bot bot merged commit 0bb862c into rapidsai:branch-24.04 Mar 5, 2024

robertmaynard deleted the fea/generate_pinned_manifest_file branch March 5, 2024 13:54

This was referenced Apr 25, 2024

WIP: Bump fmt (to 10.2.1) and spdlog (to 1.13), remove patches #592

Closed

limit pinning tests to CPM-downloaded projects #599

Merged

vyasr mentioned this pull request May 14, 2024

[BUG] Builds are not reproducible rapidsai/cudf#11533

Closed

Conversation

robertmaynard commented Jan 30, 2024

Description

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

vyasr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vyasr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vyasr Mar 1, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

robertmaynard commented Mar 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants