Deprecate the use of `LLVM_ENABLE_PROJECTS` for openmp

[petrhosek]

We should deprecate the use of LLVM_ENABLE_PROJECTS for openmp in favor of LLVM_ENABLE_RUNTIMES which would greatly simplify the build and allow improvements that are currently not possible.

[llvmbot]

@llvm/issue-subscribers-openmp

Author: Petr Hosek (petrhosek)

Details

We should deprecate the use of `LLVM_ENABLE_PROJECTS` for openmp in favor of `LLVM_ENABLE_RUNTIMES` which would greatly simplify the build and allow improvements that are currently not possible.

[ms178]

Speaking as an end user of LLVM, changing to the runtimes method would be a quality of life regression.

1. One main disadvantage is that the build process could fail very late with the LLVM_ENABLE_RUNTIMES method. This wasted a lot of time in the past, especially when building with FullLTO (also due to 2.), as with LLVM_ENABLE_PROJECTS the build would fail far earlier.

2. The runtimes method also proved to be less reliable in general at that time, especially when using LTO.

3. The method was also less flexible in the choice of linkers, as it mandated the use of the just-built lld.

Maybe there has been some work in the area to mitigate these points? It has been a couple of years since I last experimented with both methods and never had a reason to come back to re-evaluate this choice.

[jhuber6]

Speaking as an end user of LLVM, changing to the runtimes method would be a quality of life regression.

1. One main disadvantage is that the build process could fail very late with the `LLVM_ENABLE_RUNTIMES` method. This wasted a lot of time in the past, especially when building with FullLTO (also due to 2.), as with `LLVM_ENABLE_PROJECTS` the build would fail far earlier.

One issue I've had is that the compiler-rt project forces builds to take minutes longer even on trivial rebuilds #98897. Without that, most of the extra time is configuring multiple CMake projects.

2. The runtimes method also proved to be less reliable in general at that time, especially when using LTO.

Can you expand on this?

3. The method was also less flexible in the choice of linkers, as it mandated the use of the just-built lld.

Maybe there has been some work in the area to mitigate these points? It has been a couple of years since I last experimented with both methods and never had a reason to come back to re-evaluate this choice.

You should be able to pass some arguments like -DRUNTIMES_CMAKE_ARGS="-DCMAKE_CXX_COMPILER_FLAGS=-fuse-ld=bfd" or something, never tried it but there are ways to hack things through.

I think I really need to write some documentation on how to use the offloading infrastructure for things like multilibs, cross-compiling, passing arguments to the runtimes job, etc.

[shiltian]

I think a lot of uses in the openmp community, especially those only care about libomp instead of its offloading functionality, would find it inconvenient. I'm personally not a fan of using runtime build as well, even though I enabled that for openmp.

One of the biggest disadvantages is, we now have to use very tedious and complicated approach to pass CMake arguments to the runtime build, and it seems like (if I use it correctly) that there is no way to set some CMake arguments for a certain runtime. For example, when I was working on openmp, I typically build release LLVM with debug openmp. It is still doable but I need to use RUNTIMES_CMAKE_ARGS. What if I work on multiple runtimes, but only want to pass -DCMAKE_BUILD_TYPE=Debug to openmp, but not the others?

Another inconvenience is the compile database, even though @jhuber6 made some efforts in the past, and that was reverted.

[ms178]

> > ```
> > 2. The runtimes method also proved to be less reliable in general at that time, especially when using LTO.
> > ```
> 
> Can you expand on this?

I can remember that we've worked together on quite a few issues that I've reported in that time frame (2022) where that exact question came up in #55446 (comment)

1. Build error: '__nvvm_bar_warp_sync' needs target feature ptx60|ptx61|ptx63|ptx64|ptx65|ptx70|ptx71|ptx72|ptx73|ptx74|ptx75 #55446

2. Build issue in OpenMP runtime with -Xclang -Rpass-analysis=polly #56692

3. OpenMP build issue with error: version script assignment of 'VERSION' to symbol #59660

4. Not one of mine, but also a good example: llvm build failure for openmp #59537

And while trying it out, OpenMP broke way too often using that method. Hence I quickly went back to the proven method to save me some time and headaches. :)

I also got this comment in which doesn't provide confidence for power users like me that want to use LTO (at least on the other parts that work fine with it).

We generally don't test host OpenMP with LTO enabled. We can only leave this open. Thanks for the report.

That particular issue (#64217) is also still unresolved to this date.

> You should be able to pass some arguments like -DRUNTIMES_CMAKE_ARGS="-DCMAKE_CXX_COMPILER_FLAGS=-fuse-ld=bfd" or something, never tried it but there are ways to hack things through.

As these arguments are also not part of any default distribution builds and the docs were generally not as detailed (especially the OpenMP docs at that time), this would indeed need some documentation. A way to exclude LTO only for openmp (if neccessary) would be great to get more fine grained control.

[jhuber6]

Another inconvenience is the compile database, even though @jhuber6 made some efforts in the past, and that was reverted.

It's still there, I haven't noticed any problems.

[petrhosek]

I think there's some confusion between the runtimes build and bootstrapping build.

The runtimes build (rooted at runtimes/CMakeLists.txt) is used to build runtimes specified by LLVM_ENABLE_RUNTIMES with whatever host compiler you specify by CMAKE_{C,CXX}_COMPILER and other options you set for your runtime.

The bootstrapping build (rooted at llvm/CMakeLists.txt) is used to first build a toolchain with projects specified by LLVM_ENABLE_PROJECTS, and then uses that just built toolchain to build selected runtimes using the runtimes build.

You'd use LLVM_ENABLE_RUNTIMES to select which runtimes even in the bootstrapping build and which is likely where the confusion comes from. Since you mentioned having to rebuild the full toolchain just to build openmp, it sounds like you were using the bootstrapping build while the runtimes build would have been a better fir for your use case.

libc++ which has deprecated the LLVM build in favor of runtimes build some time ago has great documentation with more details at https://libcxx.llvm.org/VendorDocumentation.html

If you're wondering why we're trying to deprecate the use of LLVM build (that is LLVM_ENABLE_PROJECTS) in favor runtimes build, it's because LLVM build is designed and optimized for building LLVM and LLVM-based tools like Clang; runtime libraries like libc++, libc, compiler or openmp are not LLVM-based tools though, they're libraries that share API/ABI with LLVM.

LLVM build sets flags that are actually undesirable for runtime libraries; we're often working around it inside each runtime, but that only makes everything more complicated. Having a separate build for runtime libraries lets us sets the right flags without having to fight the LLVM build.

[ms178]

I think there's some confusion between the runtimes build and bootstrapping build.

The runtimes build (rooted at runtimes/CMakeLists.txt) is used to build runtimes specified by LLVM_ENABLE_RUNTIMES with whatever host compiler you specify by CMAKE_{C,CXX}_COMPILER and other options you set for your runtime.

The bootstrapping build (rooted at llvm/CMakeLists.txt) is used to first build a toolchain with projects specified by LLVM_ENABLE_PROJECTS, and then uses that just built toolchain to build selected runtimes using the runtimes build.

You'd use LLVM_ENABLE_RUNTIMES to select which runtimes even in the bootstrapping build and which is likely where the confusion comes from. Since you mentioned having to rebuild the full toolchain just to build openmp, it sounds like you were using the bootstrapping build while the runtimes build would have been a better fir for your use case.

No, I was not speaking of a bootstrap build. To clarify, these are my current LLVM cmake options:

    cmake -S llvm -B build -G Ninja \
        -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_INSTALL_PREFIX=/usr \
        -DLLVM_BINUTILS_INCDIR=/usr/include \
        -DFFI_INCLUDE_DIR=$(pkg-config --variable=includedir libffi) \
        -DLLVM_HOST_TRIPLE=$CHOST \
        -DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-pc-linux-gnu" \
        -DLLVM_BUILD_RUNTIME=ON \
        -DLLVM_BUILD_LLVM_DYLIB=ON \
        -DLLVM_LINK_LLVM_DYLIB=ON \
        -DCLANG_LINK_CLANG_DYLIB=ON \
        -DLLVM_DYLIB_COMPONENTS="all" \
        -DLLVM_INSTALL_UTILS=ON \
        -DLLVM_ENABLE_RTTI=ON \
        -DLLVM_ENABLE_FFI=ON \
        -DLLVM_ENABLE_BINDINGS=OFF \
        -DCMAKE_C_FLAGS="$CFLAGS -g0 -Wp,-D_FORTIFY_SOURCE=0" \
        -DCMAKE_CXX_FLAGS="$CXXFLAGS -g0 -Wp,-D_FORTIFY_SOURCE=0" \
        -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" \
        -DCMAKE_MODULE_LINKER_FLAGS="$LDFLAGS" \
        -DCMAKE_SHARED_LINKER_FLAGS="$LDFLAGS" \
        -DLLVM_INCLUDE_BENCHMARKS=OFF \
        -DLLVM_INCLUDE_TESTS=OFF \
        -DLLVM_INCLUDE_EXAMPLES=OFF \
        -DLLVM_BUILD_DOCS=OFF \
        -DLLVM_INCLUDE_DOCS=OFF \
        -DLLVM_ENABLE_OCAMLDOC=OFF \
        -DLLVM_ENABLE_SPHINX=OFF \
        -DLLVM_ENABLE_DOXYGEN=OFF \
        -DLLVM_POLLY_LINK_INTO_TOOLS=ON \
        -DLLVM_TARGETS_TO_BUILD="AMDGPU;X86;BPF" \
        -DLLVM_ENABLE_PROJECTS="clang;lld;polly;openmp;compiler-rt" \
        -DLLVM_ENABLE_ZLIB=ON \
        -DLLVM_ENABLE_ZSTD=ON \
        -DLIBCLANG_BUILD_STATIC=ON \
        -DLLVM_ENABLE_Z3_SOLVER=ON \
        -DLIBOMP_INSTALL_ALIASES=ON \
        -DLIBOMP_ENABLE_SHARED=ON \
        -DOPENMP_ENABLE_LIBOMP_PROFILING=OFF \
        -DLIBOMP_OMPT_SUPPORT=ON \
        -DLIBOMP_OMPD_SUPPORT=ON \
        -DLLVM_INSTALL_TOOLCHAIN_ONLY=OFF \
        -DCLANG_DEFAULT_PIE_ON_LINUX=OFF

    ninja -C build

The only difference was adding a line for "-DLLVM_ENABLE_RUNTIMES=openmp;compiler-rt" and deleting openmp and compiler-rt from the Projects line.

libc++ which has deprecated the LLVM build in favor of runtimes build some time ago has great documentation with more details at https://libcxx.llvm.org/VendorDocumentation.html

If you're wondering why we're trying to deprecate the use of LLVM build (that is LLVM_ENABLE_PROJECTS) in favor runtimes build, it's because LLVM build is designed and optimized for building LLVM and LLVM-based tools like Clang; runtime libraries like libc++, libc, compiler or openmp are not LLVM-based tools though, they're libraries that share API/ABI with LLVM.

LLVM build sets flags that are actually undesirable for runtime libraries; we're often working around it inside each runtime, but that only makes everything more complicated. Having a separate build for runtime libraries lets us sets the right flags without having to fight the LLVM build.

Thank you for the detailed clarification and for explaining the distinction between the runtimes build and bootstrapping. I appreciate the context around why the LLVM build (LLVM_ENABLE_PROJECTS) is suboptimal for runtime libraries like OpenMP, and I agree that aligning flags and avoiding workarounds for runtimes is a worthwhile goal.

To ensure a smooth transition for users like myself, could you elaborate on how to address the historical pain points?

Specifically:

1. Build Reliability: Is there a clever way to mitigate the risk of late build failures due to the runtime build running after the rest of the toolchain has been built? Are there recommended practices to ensure early validation of flags/toolchain compatibility before investing time in a full build of the runtimes?

2. Linker Flexibility: The documentation suggests the runtimes build respects host toolchain settings. Does this mean users can now freely choose their linker (e.g., GNU ld, gold, or lld) without being forced into the just-built lld? This would resolve a major inflexibility concern.

3. Workflow Integration: Is there anything distro maintainers or users bundling LLVM/Clang with runtimes like OpenMP need to be aware of, or is the change just as simple as the one line change in the configuration that I mentioned inline above?

If the runtimes build now offers greater reliability and flexibility while avoiding the technical debt of the LLVM build’s workarounds, I’m open to adopting it. However, clear migration guidance (e.g., concrete examples for OpenMP, handling cross-compilation, or LTO configurations) would help users like myself transition confidently. The libc++ documentation is a great reference—would similar guidelines be available for OpenMP or other runtimes?

Lastly, is there testing/CI coverage to validate the robustness of the runtimes build with LTO and alternative linkers? This would alleviate concerns about build regressions from earlier experiences. Thanks again for your patience and insights!