Skip to content

DRAFT: OpenMP Target Build Scripts for El Cap #1937

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

DRAFT: OpenMP Target Build Scripts for El Cap #1937

wants to merge 5 commits into from

Conversation

@rchen20
Copy link
Member

@rchen20 rchen20 commented Nov 4, 2025

Summary

  • This PR is a feature
  • It does the following:
    • Adds OpenMP Target build scripts for the El Cap platform at the request of me

@rchen20 rchen20 self-assigned this Nov 4, 2025
@rchen20
Copy link
Member Author

rchen20 commented Nov 4, 2025

The following linking error occurs:

[ 21%] Linking CXX executable ../../../test-forall-atomic-basic-OpenMPTarget.exe
cd /p/vast1/chen59/allraja/rajaomptarget/raja_git_omptargetelcap/build_lc_toss4-amdclang-omptarget-6.4.2-gfx942/test/functional/forall/atomic-basic && /usr/tce/backend/installations/linux-rhel8-x86_64/gcc-8.3.1/cmake-3.24.2-ywx52e32uh6gkxzuyubpwkulzgdvxyh6/bin/cmake -E cmake_link_script CMakeFiles/test-forall-atomic-basic-OpenMPTarget.exe.dir/link.txt --verbose=1
/opt/rocm-6.4.2/llvm/bin/amdclang++  -Wall -Wextra      --gcc-toolchain=/opt/rh/gcc-toolset-13/root/usr -O2 -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx942 "CMakeFiles/test-forall-atomic-basic-OpenMPTarget.exe.dir/test-forall-atomic-basic-OpenMPTarget.cpp.o" -o ../../../test-forall-atomic-basic-OpenMPTarget.exe  ../../../../lib/libgtest_main.a ../../../../lib/libgtest.a -lpthread ../../../../lib/libRAJA.a -lpthread ../../../../lib/libcamp.a -ldl 
/opt/rocm-6.4.2/lib/llvm/bin/clang-offload-packager: warning: Multiple inputs match to a single file, '/var/tmp/chen59/libRAJA-openmp-amdgcn-amd-amdhsa-unpackaged-f17783.bc'
lld: error: undefined symbol: memcpy
>>> referenced by /var/tmp/chen59/libcamp-openmp-amdgcn-amd-amdhsa-2f0fcc.o:(__omp_offloading_4e_55a8c43a__ZN4RAJA6policy3omp11forall_implILm8ERNS_17TypedRangeSegmentIllEEZ25ForallAtomicBasicTestImplINS1_28omp_target_parallel_for_execILm8EEENS_11auto_atomicEN4camp9resources2v13OmpElS4_dEvT2_EUllE_NS_4expt15ForallParamPackIJEEEEENSt9enable_ifIXsr6all_ofINSG_11type_traits18is_ForallParamPackISE_EENSK_24is_ForallParamPack_emptyISE_EEEE5valueENSC_10EventProxyISD_EEE4typeESD_RKNS7_IXT_EEEOT0_OT1_SE__l137)
>>> referenced by /var/tmp/chen59/libcamp-openmp-amdgcn-amd-amdhsa-2f0fcc.o:(__omp_offloading_4e_55a8c43a__ZN4RAJA6policy3omp11forall_implILm8ERNS_17TypedRangeSegmentIllEEZ25ForallAtomicBasicTestImplINS1_28omp_target_parallel_for_execILm8EEENS_11auto_atomicEN4camp9resources2v13OmpElS4_dEvT2_EUllE_NS_4expt15ForallParamPackIJEEEEENSt9enable_ifIXsr6all_ofINSG_11type_traits18is_ForallParamPackISE_EENSK_24is_ForallParamPack_emptyISE_EEEE5valueENSC_10EventProxyISD_EEE4typeESD_RKNS7_IXT_EEEOT0_OT1_SE__l137)
>>> referenced by /var/tmp/chen59/libcamp-openmp-amdgcn-amd-amdhsa-2f0fcc.o:(__omp_offloading_4e_55a8c43a__ZN4RAJA6policy3omp11forall_implILm8ERNS_17TypedRangeSegmentIllEEZ25ForallAtomicBasicTestImplINS1_28omp_target_parallel_for_execILm8EEENS_11auto_atomicEN4camp9resources2v13OmpElS4_dEvT2_EUllE_NS_4expt15ForallParamPackIJEEEEENSt9enable_ifIXsr6all_ofINSG_11type_traits18is_ForallParamPackISE_EENSK_24is_ForallParamPack_emptyISE_EEEE5valueENSC_10EventProxyISD_EEE4typeESD_RKNS7_IXT_EEEOT0_OT1_SE__l137)
>>> referenced 969 more times
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

After consultation, John G. found in his own tests that this error occurs when a memcpy occurs within an OpenMP Target region, which shouldn't be allowed within such a region. @trws or @MrBurmark would you have any insight to how this might happen?

@MrBurmark
Copy link
Member

MrBurmark commented Nov 4, 2025

It looks like this occurs within an atomic test. This might come from use of memcpy in the implementation of atomics for types that are not natively supported.

rhornung67
rhornung67 reviewed Nov 4, 2025
View reviewed changes
scripts/lc-builds/toss4_amdclang_omptarget.sh
-DHIP_ROOT_DIR="/opt/rocm-${COMP_VER}/hip" \
-DHIP_PATH=/opt/rocm-${COMP_VER}/llvm/bin \
-DENABLE_CLANGFORMAT=On \
-DCLANGFORMAT_EXECUTABLE=/opt/rocm-5.2.3/llvm/bin/clang-format \
Copy link
Member

@rhornung67 rhornung67 Nov 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should point at regular clang for clang-format

rhornung67
rhornung67 reviewed Nov 4, 2025
View reviewed changes
host-configs/lc-builds/toss4/hip_3_X.cmake
set(CMAKE_CXX_FLAGS_RELEASE "-O2" CACHE STRING "")
set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "-O2 -g" CACHE STRING "")
set(CMAKE_CXX_FLAGS_DEBUG "-O0 -g" CACHE STRING "")
set(CMAKE_CXX_FLAGS_RELEASE "--gcc-toolchain=/opt/rh/gcc-toolset-13/root/usr -O2" CACHE STRING "")
Copy link
Member

@rhornung67 rhornung67 Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be using the hip_4 host-config file

@rhornung67
Copy link
Member

rhornung67 commented Nov 4, 2025

@rchen20 does the regular clang compiler work?

Looking through the code, I don't see anything obviously wrong. Could the memcpy be related to the camp EventProxy?

@rchen20
Copy link
Member Author

rchen20 commented Nov 4, 2025

@rchen20 does the regular clang compiler work?

The regular clang compiler is not available on our El Cap platforms, so I should remove that particular script.

Looking through the code, I don't see anything obviously wrong. Could the memcpy be related to the camp EventProxy?

I don't see any memcpy's in Camp::EventProxy, but there are implementations of memcpy in Camp::resource. After removing those implementations, the same memcpy linking error occurs. I may need to do as Jason suggests and dig in to our atomic implementations and see if the memcpy is coming from there.

@rhornung67
Copy link
Member

rhornung67 commented Nov 4, 2025

@rchen20 I skimmed the RAJA atomics and did not see any memcpy calls. A thought....what happens if you remove the CAS atomic check here https://github.com/LLNL/RAJA/blob/develop/test/functional/forall/atomic-basic/tests/test-forall-atomic-basic.hpp#L105?

@rchen20
Copy link
Member Author

rchen20 commented Nov 4, 2025

@rchen20 I skimmed the RAJA atomics and did not see any memcpy calls. A thought....what happens if you remove the CAS atomic check here https://github.com/LLNL/RAJA/blob/develop/test/functional/forall/atomic-basic/tests/test-forall-atomic-basic.hpp#L105?

That's a good idea, but it still gave the same error after removing the CAS operation.

@rhornung67
Copy link
Member

rhornung67 commented Nov 4, 2025

Maybe try eliminating entries in the list of data types we test to see if that can help pin down the issue?

rchen20
rchen20 commented Nov 12, 2025
View reviewed changes
include/RAJA/util/TypeConvert.hpp
Comment on lines +36 to +47
void * custom_memcpy( void * dest, const void * src, size_t len )
{
char * customdest = (char *) dest;
const char * customsrc = (const char *) src;

for ( size_t ii = 0; ii < len; ++ii )
{
customdest[ii] = customsrc[ii];
}

return dest;
}
Copy link
Member Author

@rchen20 rchen20 Nov 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't need to do this. Asking AMD about this.

@rchen20 rchen20 mentioned this pull request Nov 14, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rhornung67 rhornung67 rhornung67 left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@rchen20 rchen20

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rchen20 @MrBurmark @rhornung67