Remove special handling for dialect in Thrust's build system. by alliepiper · Pull Request #6722 · NVIDIA/cccl

alliepiper · 2025-11-21T00:27:54Z

Closes #6573

This ended up including several rabbit holes from knock-on effects:

The python CCCL install relied on CMake variables that are not provided by find_package, and just happened to be available for reasons. The refactorings in this PR removed those variables and broke the install.
Fixed the python wheel build CMake to reuse CCCL's existing install rules instead of rolling their own with unreliable state.
Fixed the linux python wheel builds to setup the compilers consistently for CCCL.
Fixed the windows python wheel builds to always use Ninja, which always sets up compilers consistently and applies our desired build flags.
Added some python build artifacts to .gitignore files discovered after testing the python wheel build changes.
Discovered that some CUB headers were not being installed due to not following CUB header naming conventions. Fixed these.

copy-pr-bot · 2025-11-21T00:27:57Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

alliepiper · 2025-11-21T00:40:13Z

/ok to test

alliepiper · 2025-11-21T00:41:41Z

cmake/CCCLBuildCompilerTargets.cmake

-  # detect that the following instructions are unreachable. This is intentional
-  # and unavoidable in these cases. This target can be used to silence
-  # unreachable code warnings.
-  add_library(cccl.silence_unreachable_code_warnings INTERFACE)


This is no longer used, drive-by cleanup.

alliepiper · 2025-11-21T00:42:33Z

cub/CMakeLists.txt

-enable_language(CXX)
-
-# Support adding CUB to a parent project via add_subdirectory.
-# See examples/cmake/add_subdir/CMakeLists.txt for details.


Made this cleanup across several projects -- these comments refer to hacks we no longer rely on, and the overall pattern can be cleaned up significantly as shown.

alliepiper · 2025-11-21T00:43:15Z

cub/CMakeLists.txt

-if (CUB_ENABLE_TESTING OR CUB_ENABLE_EXAMPLES)
-  include(CTest)
-  enable_testing()
-endif()


Made this refactoring across several projects. We always enable this in the CCCL root, so these have no effect.

alliepiper · 2025-11-21T00:44:01Z

libcudacxx/cmake/LibcudacxxInternalHeaderTesting.cmake


 function(libcudacxx_create_internal_header_test header_name headertest_src)
  # Create the default target for that file
-  set(internal_headertest_${header_name} verify_${header_name})


Drive-by clean-up -- this has no effect, the created variable is never used. Same for the other libcu++ header test files.

alliepiper · 2025-11-21T00:53:55Z

/ok to test

alliepiper · 2025-11-21T01:55:27Z

thrust/cmake/ThrustHeaderTesting.cmake

  )
  list(REMOVE_ITEM headers ${headers_exclude_details})

-  # List of headers that aren't implemented for all backends, but are implemented for CUDA.


This block is a leftover from thrust::async:: test special handling. No longer needed.

alliepiper · 2025-11-21T02:01:13Z

/ok to test

alliepiper · 2025-11-21T12:54:17Z

/ok to test

This enables python to build the c libraries it depends on.

The old implementation reinvented the wheel and depended on variables that can't be relied upon.

alliepiper · 2025-11-21T14:43:05Z

/ok to test

alliepiper · 2025-11-21T14:47:35Z

ci/windows/build_cuda_cccl_python.ps1

    **Required.** The Python version to use for building the wheel, expressed
    as `<major>.<minor>` (e.g. `3.11`).

-.PARAMETER UseNinja


Removed the ability to use anything but ninja here. The default VS generator doesn't add all of the flags we need, and ninja is always available in our devcontainers.

This was a knock-on effect of the wheel packaging refactor that was necessary to fix how python installs CCCL headers.

alliepiper · 2025-11-21T14:48:47Z

ci/build_cuda_cccl_wheel.sh

 cuda_version=$(nvcc --version | grep -oP 'release \K[0-9]+\.[0-9]+' | cut -d. -f1)
 echo "Detected CUDA version: ${cuda_version}"

+# Configure compilers:


Required now that python is properly installing our headers without relying on CMake variables that aren't reliable.

alliepiper · 2025-11-21T14:52:17Z

python/cuda_cccl/CMakeLists.txt

  "Building for CUDA ${CUDA_VERSION_MAJOR}, output directory: ${CUDA_VERSION_DIR}"
 )

+# Build cccl.c.parallel and add CCCL's install rules


Refactored to reuse CCCL's existing install rules, and avoid using <subproject>_SOURCE_DIR vars, which are not provided by find_package and only happened to be defined here randomly. These variables are no longer available here after other refactorings in this PR, so fixing this now make sense.

alliepiper · 2025-11-21T15:11:49Z

/ok to test

alliepiper · 2025-11-21T15:13:55Z

cub/cub/detail/ptx-json/array.cuh

 #pragma once

-#include <cub/detail/ptx-json/object.h>
+#include <cub/detail/ptx-json/object.cuh>


Another knock-on effect of fixing python's CCCL install, it exposed that our installs were missing some CUB headers due to some headers not following conventions. CUB's headers end in .cuh, not .h

We use .cuh, not .h in CUB. This broke our install rules.

alliepiper · 2025-11-21T15:26:21Z

/ok to test

bernhardmgruber · 2025-11-21T16:10:43Z

Discovered that some CUB headers were not being installed due to not following CUB header naming conventions. Fixed these.

This may actually be on purpose. I think I saw that the extension of some JSON headers were changed. Those are only needed when CUB is compiled from CCCL.C, and will hopefully go away over time when we roll out the new tuning API design. I think those headers are not meant to be installed or packaged with the CTK. @wmaxey and @griwes may know more here.

bernhardmgruber

Except for the change of file extensions, which I am unsure about, LGTM! It's great to see we could remove so much logic! Thx a lot!

alliepiper · 2025-11-21T16:32:05Z

Gotcha. If that's the case we should exclude them from the install, but keep the header conventions.

I checked and they aren't included anywhere other than the c libraries and cub, and the cub usages are gated behind a macro, so installing them shouldn't hurt anything. Fixing the extensions did break the header tests since they're being picked up for that now, but I'm fixing that now, commit incoming!

alliepiper · 2025-11-21T16:41:57Z

/ok to test

github-actions · 2025-11-21T18:55:11Z

🥳 CI Workflow Results

🟩 Finished in 2h 10m: Pass: 100%/267 | Total: 2d 00h | Max: 51m 34s | Hits: 99%/377961

See results here.

shwina

Python CMakeLists.txt changes and build script changes look good.

…#6722) * Remove special handling for dialect in Thrust's build system. * Allow consumers to set CCCL_TOP_LEVEL_PROJECT. This enables python to build the c libraries it depends on. * Force windows python builds to use ninja. * Exclude python build artifacts from git. * Fix python build to reuse CCCL's existing install rules. The old implementation reinvented the wheel and depended on variables that can't be relied upon. * Fix CUB header extensions. We use .cuh, not .h in CUB. This broke our install rules. * Fixup ptx_json header testing.

* Improvements to inspect_changes CI functionality. (#6535) 1. Rewrote `inspect_changes.sh` to python. 2. Split out project name, path, dependency information into new yaml file. 3. Simplified dependency specification (only direct dependencies are needed). 4. Split dependency specification into two types: full (use the pull_request matrix) and lite (use the pull_request_lite matrix). 5. Use new features to split some projects with expensive dependency chains into 'public' (public headers, etc) and `internal` (tests/examples/infra, etc). Dependencies are only added when the public API files change. 6. Update the ignored paths to include newer additions. 7. Add tests for inspect_changes to make it easier to test and validate modifications. * Add deps on thrust/cub to libcudacxx. (#6694) Complete the circle. * Fix oversubscription issue with lit precompile, label hack (#6554) * Remove MSVC hint that is applied globally in CCCL. * Fix oversubscription issue with lit precompile, label hack. Moves the C2H tests earlier in the config so we can add the fake dependency to lit. Explains and labels the hack we're using to avoid oversubscription. * Make missing sccache nonfatal. (#6582) * Add nvbench_helper tests to CI. (#6679) * Drive-by fix to packaging test script. Building isn't needed at the moment, but this will save some headaches if we add any executables that need to be built to this preset. * Fix warnings in nvbench_helper tests. * Skip test sizes that OOM CI runners. * Update boost dep to work with CMake 4+ * Enable CI coverage for nvbench_helper tests. * Update inspect_changes smoke tests. * Update libcudacxx C++ dialect handling. (#6693) * Switch preprocessor cache to S3. (#6561) This is more robust than the github cache approach, which evicts caches regularly due to a 10GB repo limit. The S3 cache will be more reliable and persistent. This also allows preprocessor caching in our linux devcontainers, improving developer experience with faster build times. * Restore libcudacxx dialect presets. (#6705) These were removed in #6693. * Remove special dialect handling from cudax build system. (#6702) * Remove special handling of C++ dialect in CUB's build system. (#6713) * Remove special handling for dialect in Thrust's build system. (#6722) * Remove special handling for dialect in Thrust's build system. * Allow consumers to set CCCL_TOP_LEVEL_PROJECT. This enables python to build the c libraries it depends on. * Force windows python builds to use ninja. * Exclude python build artifacts from git. * Fix python build to reuse CCCL's existing install rules. The old implementation reinvented the wheel and depended on variables that can't be relied upon. * Fix CUB header extensions. We use .cuh, not .h in CUB. This broke our install rules. * Fixup ptx_json header testing. * Fix issue with libcudacxx header tests. (#6785) Did some drive-by cleanups/standardizing of how our internal libraries link together. * Improve CMake package handling, add MSVC compat flags to libcudacxx's public interface. (#6791) * Change the MSVC preprocessor check to an error. When `/Zc:preprocessor` is not set, the build will error out before `#pragma message` directives are handled. Changing to an error ensures that this will be caught. * Add /Zc:preprocessor and /Zc:__cplusplus to public libcudacxx::libcudacxx target. * Remove dev build defs for /Zc:preprocessor and /Zc:__cplusplus. * Recheck languages on successive `find_package(libcudacxx)` calls. * Clean up dependency handling between projects. * Set <project>_DIR when locating packages with our add_subdir helper. This ensures that it behaves the same as find_package, and that later calls to find_package will locate the same configs. * Update test_packaging.sh to use the local repo for CPM. * Enable CUDA language and verbose logging for CMake example configs. * Reduce verbosity of libcudacxx package. * Fix early return checks. * Improve some option names / diagnostics. * Remove check for unsupported CTK versions.

github-project-automation bot added this to CCCL Nov 21, 2025

github-project-automation bot moved this to Todo in CCCL Nov 21, 2025

cccl-authenticator-app bot moved this from Todo to In Progress in CCCL Nov 21, 2025

alliepiper commented Nov 21, 2025

View reviewed changes

alliepiper force-pushed the thrust-dialect-handling branch from a199c69 to 0fcc37a Compare November 21, 2025 00:46

alliepiper commented Nov 21, 2025

View reviewed changes

alliepiper force-pushed the thrust-dialect-handling branch from 0fcc37a to 8444963 Compare November 21, 2025 02:01

This comment has been minimized.

Sign in to view

Remove special handling for dialect in Thrust's build system.

64ae6e6

alliepiper force-pushed the thrust-dialect-handling branch from 8444963 to 64ae6e6 Compare November 21, 2025 12:54

This comment has been minimized.

Sign in to view

alliepiper added 4 commits November 21, 2025 14:36

Allow consumers to set CCCL_TOP_LEVEL_PROJECT.

41112f7

This enables python to build the c libraries it depends on.

Force windows python builds to use ninja.

0eb3479

Exclude python build artifacts from git.

8dd4f7e

Fix python build to reuse CCCL's existing install rules.

39c4d23

The old implementation reinvented the wheel and depended on variables that can't be relied upon.

alliepiper commented Nov 21, 2025

View reviewed changes

Fix CUB header extensions.

1237e47

We use .cuh, not .h in CUB. This broke our install rules.

alliepiper force-pushed the thrust-dialect-handling branch from bc9a654 to 1237e47 Compare November 21, 2025 15:26

bernhardmgruber approved these changes Nov 21, 2025

View reviewed changes

Fixup ptx_json header testing.

38172b6

This comment has been minimized.

Sign in to view

alliepiper marked this pull request as ready for review November 21, 2025 19:30

alliepiper requested review from a team as code owners November 21, 2025 19:30

alliepiper requested review from gonidelis, griwes and jrhemstad November 21, 2025 19:30

cccl-authenticator-app bot moved this from In Progress to In Review in CCCL Nov 21, 2025

shwina approved these changes Nov 21, 2025

View reviewed changes

alliepiper merged commit c54b2a6 into NVIDIA:main Nov 21, 2025
554 of 559 checks passed

github-project-automation bot moved this from In Review to Done in CCCL Nov 21, 2025

alliepiper deleted the thrust-dialect-handling branch November 21, 2025 20:28

Conversation

alliepiper commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot bot commented Nov 21, 2025

Uh oh!

alliepiper commented Nov 21, 2025

Uh oh!

alliepiper Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

alliepiper Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

alliepiper Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alliepiper Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alliepiper commented Nov 21, 2025

Uh oh!

alliepiper Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

alliepiper commented Nov 21, 2025

Uh oh!

This comment has been minimized.

alliepiper commented Nov 21, 2025

Uh oh!

This comment has been minimized.

alliepiper commented Nov 21, 2025

Uh oh!

alliepiper Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

alliepiper Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

alliepiper Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alliepiper commented Nov 21, 2025

Uh oh!

alliepiper Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

alliepiper commented Nov 21, 2025

Uh oh!

bernhardmgruber commented Nov 21, 2025

Uh oh!

bernhardmgruber left a comment

Choose a reason for hiding this comment

Uh oh!

alliepiper commented Nov 21, 2025

Uh oh!

alliepiper commented Nov 21, 2025

Uh oh!

This comment has been minimized.

github-actions bot commented Nov 21, 2025

🥳 CI Workflow Results

🟩 Finished in 2h 10m: Pass: 100%/267 | Total: 2d 00h | Max: 51m 34s | Hits: 99%/377961

Uh oh!

shwina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alliepiper commented Nov 21, 2025 •

edited

Loading

alliepiper Nov 21, 2025 •

edited

Loading

alliepiper Nov 21, 2025 •

edited

Loading

alliepiper Nov 21, 2025 •

edited

Loading