add a CUDA device code sanity check#4692

Merged

boegel merged 120 commits intoeasybuilders:developfrom

jfgrimm:cuda-device-code-sanity-check

May 20, 2025

Commits on Oct 24, 2024

sanity check binaries/libraries for device code matching cuda_compute_capabilities when CUDA is used
jfgrimm
committed

Commits on Feb 19, 2025

Merge branch '5.0.x' into cuda-device-code-sanity-check
casparvl
authored

Commits on Feb 20, 2025

Add check for PTX, more explicit debug logging
Caspar van Leeuwen
committed
That return should not be there, as it will stop the sanity check after the first non-cuda file. that's wrong
Caspar van Leeuwen
committed
Fix some logic in the PTX warning printed
Caspar van Leeuwen
committed
Add option for ignoring individual files in the CUDA sanity check
Caspar van Leeuwen
committed

Commits on Feb 21, 2025

Add strict-cuda-sanity-check option and make sure we only fail the sanity check on surpluss CUDA archs if this option is set. Otherwise, print warning
Caspar van Leeuwen
committed
This is a work in progress for creating a set of tests...
Caspar van Leeuwen
committed

Commits on Feb 22, 2025

First test working..
Caspar van Leeuwen
committed

Commits on Mar 5, 2025

Restructure logging messages a bit
Caspar van Leeuwen
committed
quote path in debug log string
Caspar van Leeuwen
committed
Added unit tests
Caspar van Leeuwen
committed
Fix hound issues
Caspar van Leeuwen
committed
flake8 compliance: add extra blank line
Caspar van Leeuwen
committed

Commits on Apr 3, 2025

Resolved conflict
Caspar van Leeuwen
committed
Make sure strict_cuda_sanity_check is in the list for the correct default (false). Add accept-ptx-as-cc-support and accept-missing-cuda-ptx options
Caspar van Leeuwen
committed
Add accept-ptx-as-cc-support and accept-missing-cuda-ptx options
Caspar van Leeuwen
committed
Fix indentation mistake
Caspar van Leeuwen
committed
Get rid of early return, which apparently BDFL doesn't like :)
Caspar van Leeuwen
committed
Fix too long line
Caspar van Leeuwen
committed
Implement small review comments
Caspar van Leeuwen
committed
Fixed typo
Caspar van Leeuwen
committed
Check for presence of cuobjdump. If sanity check isn't run, raise that to a warning message
Caspar van Leeuwen
committed
Raise missing cuobjdump to error: if CUDA root was defined, then this IS CUDA software. Not being able to run the CUDA sanity check should then be considered a failure
Caspar van Leeuwen
committed
downgrade to debug message, as the CUDA root not being defined is pretty 'common': this is the case for all builds of NON-cuda software :)
Caspar van Leeuwen
committed
Limit to a single return statement
Caspar van Leeuwen
committed
No more early returns
Caspar van Leeuwen
committed
Start adding summary data
Caspar van Leeuwen
committed

Commits on Apr 8, 2025

Add logic to deal with accept-ptx-for-cc-support and --accept-missing-cuda-ptx
Caspar van Leeuwen
committed
Further counting and summary collection
Caspar van Leeuwen
committed
Fix some hound issues
Caspar van Leeuwen
committed
First attempt at summary report
Caspar van Leeuwen
committed
Make formatting more readable
Caspar van Leeuwen
committed
Fix hound issues
Caspar van Leeuwen
committed
Fix one more hound issue
Caspar van Leeuwen
committed
Add clear distinction between ignores & failures. Also, clearly indicated when some summary statistic is _not_ considered a failure (because of selected options). Also, add suggestions on how to re…
Caspar van Leeuwen
committed
Truncate too long lines
Caspar van Leeuwen
committed
Fix linting errors
Caspar van Leeuwen
committed
Fix missing f for f-strings:
Caspar van Leeuwen
committed

Commits on Apr 9, 2025

Add option to ignore all CUDA sanity failures to not break current EasyBuild behaviour
Caspar van Leeuwen
committed
Implement an option te report, but ignore _all_ failures
Caspar van Leeuwen
committed
Fix typo and missing comma
Caspar van Leeuwen
committed

Commits on Apr 10, 2025

Replaced all num_X with len(files_x), we don't need separate counters
Caspar van Leeuwen
committed
Update easybuild/framework/easyblock.py

casparvl
and
jfgrimm
authored
Removed some forgotten num_files_X and replaced with len(files_X)
Caspar van Leeuwen
committed
Change option names
Caspar van Leeuwen
committed
Fix cuda-compute-capabilities description to be more specific that fat binaries aren't always possible
Caspar van Leeuwen
committed
Various changes from code review
Caspar van Leeuwen
committed
Replaced more occurences of cc by devcode
Caspar van Leeuwen
committed
only store relative paths in the files_X variables
Caspar van Leeuwen
committed
Processed various review comments...
Caspar van Leeuwen
committed
Fix hound issues
Caspar van Leeuwen
committed
Renamed function
Caspar van Leeuwen
committed
Various review comments processed
Caspar van Leeuwen
committed
Fixed hound issues:
Caspar van Leeuwen
committed
Make sure to raise an error if cuobjdump doesnt exist, or if it returns something very unexpected
Caspar van Leeuwen
committed
Raise info to warning when we're not erroring on failure
Caspar van Leeuwen
committed

Commits on Apr 14, 2025

Fix linting issues
Caspar van Leeuwen
committed
Deduplicate code by replacing get_cuda_device_code_and_ptx_architectures by get_cuda_architectures. This can then be called once to retrieve contents from the elf code sections (device code archs) …
Caspar van Leeuwen
committed
Grammar fix
Caspar van Leeuwen
committed
Fix whitespace
Caspar van Leeuwen
committed
Fix undefined name
Caspar van Leeuwen
committed

Commits on Apr 15, 2025

Create mock setup for get_cuda_object_dump_raw and get_cuda_architecture
Caspar van Leeuwen
committed
Fix naming in config and put in the correct (alphabetical) place in the list
Caspar van Leeuwen
committed
Make sure archives are also checked. Libary does _not_ seem to be an official possibility for the output string, so we can ommit that
Caspar van Leeuwen
committed
Initial (working) version of a unit test for get_cuda_object_dump_raw
Caspar van Leeuwen
committed
Fix hound issues
Caspar van Leeuwen
committed
More test cases
Caspar van Leeuwen
committed
Remove a stray print statement
Caspar van Leeuwen
committed
Add remaining test cases for get_cuda_architecture and get_cuda_object_dump_raw
Caspar van Leeuwen
committed
Change if-elif-else into a nested if-else, with an if-if. This is since _BOTH_ additional_devcodes and missing_devcodes may be true, it's not either or. And we want to sanity check to report on _bo…
Caspar van Leeuwen
committed
Don't keep accumulating the fail_msg after we have logged it
Caspar van Leeuwen
committed
added f to fstring...
Caspar van Leeuwen
committed
Replace Surplus with Additional in warning
Caspar van Leeuwen
committed
Updateing toy builds. 10 test cases defined, of which 3 are implemented so far
Caspar van Leeuwen
committed

Commits on Apr 16, 2025

Architectures can be 9.0a or 10.0a now, i.e. sm_90a is a valid optimization target. See https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#gpu-feature-list
Caspar van Leeuwen
committed
Some missing f-strings and small refinements in the summary reporting to make things more clear
Caspar van Leeuwen
committed
Removed old tests and replaced them with new ones. New tests check all newly added options, as well as running the sanity check when multiple CUDA compute capabilities are defined, running with no …
Caspar van Leeuwen
committed
Fix hound issues
Caspar van Leeuwen
committed
Fix linting issues
Caspar van Leeuwen
committed
Remove f-string, as there are no placeholders in this string
Caspar van Leeuwen
committed
Fix unit test expected result
Caspar van Leeuwen
committed

Commits on May 12, 2025

Add a test that triggers the if missing_ptx_ccs: if path in ignore_file_list or ignore_failrues: logic
Caspar van Leeuwen
committed
Remove blank line
Caspar van Leeuwen
committed
Now make sure test 1.a. actually fails on the test case @ocaisa found, which is a binary with correct dev code, missing ptx code, an all default args
Caspar van Leeuwen
committed
Fix the test so that it now passes once the issue is fixed... It was checking the wrong regex
Caspar van Leeuwen
committed
Fix the issue with the missing ignore_msg
Caspar van Leeuwen
committed
Apply suggestions from code review

casparvl
and
ocaisa
authored
Remove check_cuobjdump, as it is not needed anymore
Caspar van Leeuwen
committed
Format file lists on separate lines for better readability of the logs
Caspar van Leeuwen
committed
Still need to do some formatting, but things now go to trace output for more visibility
Caspar van Leeuwen
committed

Commits on May 13, 2025

Print both to trace output (with short version of advice), and to log, so that it's also preserved in the logfile
Caspar van Leeuwen
committed
Fix hound issues - and hopefully CI checks
Caspar van Leeuwen
committed
Fix missing condition
Caspar van Leeuwen
committed
Added missing f to f-string
Caspar van Leeuwen
committed
Modify tests for the new syntax, and to also check the trace output
Caspar van Leeuwen
committed
Fix hound issues
Caspar van Leeuwen
committed

Commits on May 14, 2025

Apply suggestions from code review
ocaisa
authored

Commits on May 15, 2025

Apply suggestions from code review
ocaisa
authored
Apply suggestions from code review
ocaisa
authored
Apply suggestions from code review
ocaisa
authored
Update easybuild/framework/easyblock.py

casparvl
and
ocaisa
authored
Update easybuild/framework/easyblock.py

casparvl
and
ocaisa
authored
Fix unit tests for the new setup where we check if CUDA is a dep, instead of checking that EBROOTCUDA is set
Caspar van Leeuwen
committed
Don't set EBROOTCUDA anymore, it's no longer needed
Caspar van Leeuwen
committed
Keep failure message short: just list the number of files, and refer to the build log
Caspar van Leeuwen
committed
Make defining a non-empty failure message conditional on an actual failure having occured
Caspar van Leeuwen
committed
We don't need to track with is_failure, we can just check if any of the *_fails file lists is non-empty
Caspar van Leeuwen
committed
Fix too long line
Caspar van Leeuwen
committed
Fix unit tests to accomodate for the difference in the error message that is raised
Caspar van Leeuwen
committed

Commits on May 16, 2025