Clarification for hermetic vs local CUDA installations

[Flamefire]

I'm reading https://openxla.org/xla/hermetic_cuda#pointing_to_cudacudnnnccl_redistributions_on_local_file_system and am not sure hot to interpret this:

My goal is to use CUDA etc. distributions already available on the cluster, i.e. installed locally.

Do I need to set either HERMETIC_CUDA_VERSION etc., LOCAL_CUDA_PATH etc. or both? I.e. does setting the former lead to downloading CUDA even with the latter set?
Similar for HERMETIC_CUDA_COMPUTE_CAPABILITIES: When does this need to be set exactly?

Additionally there are multiple mentions of:

If .bazelrc doesn't have corresponding entries and the environment variables
are not passed to bazel command, you can set them globally in your shell:

Do I need to set LOCAL_CUDA_PATH etc. in the environment, pass them to bazel via --repo_env or both?

Context: I'm building TensorFlow for a HPC cluster from source

[beckerhe]

You can still use hermetic CUDA for the compilation. Using hermetic CUDA means the Bazel will download and use the exact CUDA version that you specify for compilation and for testing. But if you deploy XLA (or Tensorflow) through a Python wheel or other means, then the binary will use the CUDA installation that is available at runtime. If you're using Wheels then XLA would fine the CUDA shared libraries that are installed in the virtual environment or fall back to system wide CUDA installation if the former was not available.

Regarding HERMETIC_CUDA_COMPUTE_CAPABILITIES. This defines for which GPU architectures Tensorflow kernels will be compiled for. By default we compile for a variety of architectures such that TF release run out of the box on the most common GPUs. But if you're compiling from source you can save compile time and binary size by just compiling for the exact GPU architecture that you need.

TLDR - I would recommend the following:

1. Set HERMETIC_CUDA_VERSION to the CUDA version that you're using on the cluster.
2. Set HERMETIC_CUDA_COMPUTE_CAPABILITIES to the SM architecture of the GPUs in your cluster.

[Flamefire]

We are building on a compute cluster, so CUDA_COMPUTE_CAPABILITIES and e.g. match=native are used as the hardware is known and fixed.

For security and reproducibility reasons we'd like to avoid downloading things during build as much as reasonably possible.

What we are currently blocked on is third_party/xla/xla/tsl/cuda/cupti_stub.cc (and others in third_party/xla/xla/backends/profiler/gpu) having a hard-coded include "third_party/gpus/cuda/extras/CUPTI/include/cupti.h" and I thought this might be solved by some setting and Bazel potentially creating symlinks in the appropriate location. IIRC I've seen that for other sources.

There seems to be a rule

cc_library(
    name = "headers",
    #hdrs = glob([
        #...
        #"include/cupti.h",
        #...
    #),
    include_prefix = "third_party/gpus/cuda/extras/CUPTI/include",
    includes = ["include/"],
    strip_include_prefix = "include",
    visibility = ["@local_config_cuda//cuda:__pkg__"],
)

But I'm note sure where exactly it is defined. I'd say it needs to pass the path to the CUPTI include, not add anything for local CUDA installations and just use #include "cupti.h" in the source.

[beckerhe]

For security and reproducibility reasons we'd like to avoid downloading things during build as much as reasonably possible.

The XLA build relies heavily on downloads. The only way to avoid downloads altogether is to use the flag --experimental_repository_cache. This allows you to download all dependencies into a single folder where you can vet them and then copy the folder over to a machine without internet access. FWIW just like all other downloads CUDA archives are verified against hashes, so it shouldn't add any reproducibility issues. In fact reproducibility issues with local CUDA installations was what made us develop that feature in the first place.

Regarding the CUPTI issue - This is the template that creates the rule: https://github.com/openxla/xla/blob/main/third_party/gpus/cuda/hermetic/cuda_cupti.BUILD.tpl

I believe if you see the hdrs attribute being commented out it means that the CUDA detection script couldn't find CUPTI.

[Flamefire]

Regarding the CUPTI issue - This is the template that creates the rule: https://github.com/openxla/xla/blob/main/third_party/gpus/cuda/hermetic/cuda_cupti.BUILD.tpl

I believe if you see the hdrs attribute being commented out it means that the CUDA detection script couldn't find CUPTI.

I see. TF seems to have it in 2 locations:

tensorflow-2.18.1/third_party/gpus/cuda/hermetic/cuda_cupti.BUILD.tpl
tensorflow-2.18.1/third_party/xla/third_party/tsl/third_party/gpus/cuda/hermetic/cuda_cupti.BUILD.tpl

What is the search procedure used to find CUPTI? Can I debug that somehow?

[beckerhe]

There is a Python script https://github.com/openxla/xla/blob/main/third_party/gpus/find_cuda_config.py which you can run.

I believe it should output cupti_library_dir and cupti_header_path if it was able to find those.

[Flamefire]

It does when setting TF_CUDA_PATHS:

cupti_include_dir: /software/CUDA/12.6.0/extras/CUPTI/include
cupti_library_dir: /software/CUDA/12.6.0/extras/CUPTI/lib64

However during the build of TF this script isn't called (added a debug output to verify), instead https://github.com/openxla/xla/blob/main/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl is used which doesn't work for cupti.

I traced the behavior to this:

• a build directory cuda_cupti is created with symlinks bin, include, lib, nvvm to LOCAL_CUDA_PATH
• the lib directory is searched for a libcupti*.so to determine the version
• The version is passed to the BUILD file creation
• if the version is empty a dummy file with commented out content is created

So judging from this using LOCAL_CUDA_PATH cannot work for cupti for 2 reasons

1. cupti is in a subfolder of LOCAL_CUDA_PATH
2. the library folder is lib64 not lib

I tried a small patch for TF 2.18.1 to address these 2 points and it seems to get further:

--- ./third_party/xla/third_party/tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl
+++ ./third_party/xla/third_party/tsl/third_party/gpus/cuda/hermetic/cuda_redist_init_repositories.bzl
@@ -91,7 +91,9 @@
 def _get_libraries_by_redist_name_in_dir(repository_ctx):
     lib_dir_path = repository_ctx.path("lib")
     if not lib_dir_path.exists:
-        return []
+        lib_dir_path = repository_ctx.path("lib64")
+        if not lib_dir_path.exists:
+            return []
     main_lib_name = _get_main_lib_name(repository_ctx)
     lib_dir_content = lib_dir_path.readdir()
     return [
@@ -246,10 +248,15 @@
 def _use_local_cuda_path(repository_ctx, local_cuda_path):
     # buildifier: disable=function-docstring-args
     """ Creates symlinks and initializes hermetic CUDA repository."""
+    if repository_ctx.name == 'cuda_cupti':
+      local_cuda_path += '/extras/CUPTI'
+      symlinks = ["include", "lib64"]
+    else:
+      symlinks = ["include", "lib", "bin", "nvvm"]
     use_local_path(
         repository_ctx,
         local_cuda_path,
-        ["include", "lib", "bin", "nvvm"],
+        symlinks,
     )
 
 def _use_local_cudnn_path(repository_ctx, local_cudnn_path):

Do you think this is the right approach? If so where should I PR this to?

Edit: Possibly related PR tensorflow/tensorflow#75866 that indicates someone tested LOCAL_CUDA_PATH before

[beckerhe]

@ybaturina is the right person to answer this question. Yulia do you know what's going wrong here?

[ybaturina]

Hi @Flamefire ,

Using local installation as a source for repository rules is not a recommended approach.

If for some reason you still want to use this method, the folder provided for LOCAL_CUDA_PATH should have bin with the binaries, lib with ELF files, include with headers, nvvm with .bc files (see here). This is made to provide the compatibility of the repo rules with the content of individual .tar.xz redistributions, so please don't merge your patch.

CUPTI headers should be in the include dir of LOCAL_CUDA_PATH. I understand that system-wide CUDA installation has a different path (e.g. CUDA/12.6.0/extras/CUPTI/include), but like I said, using LOCAL_CUDA_PATH is not a recommended approach, and the symlinks should be created in LOCAL_CUDA_PATH, not in the repository rule implementation.

tensorflow/tensorflow#75866 was related to a different problem (absence of NVIDIA UMD in LOCAL_CUDA_PATH).

[Flamefire]

Hi @ybaturina ,
thanks for the prompt reply.

Using local installation as a source for repository rules is not a recommended approach.

The "local installation" in this case is a fixed version as defined in a "description" file (EasyConfig), comparable to e.g. a Docker recipe. That CUDA installation is used for all other software build in this environment so it seems to be the safest option to use that for TF too, at least to avoid duplicated downloads.
This avoids reproducibility issues. Is there another reason why this is not recommended?

I understand that system-wide CUDA installation has a different path (e.g. CUDA/12.6.0/extras/CUPTI/include), but like I said, using LOCAL_CUDA_PATH is not a recommended approach, and the symlinks should be created in LOCAL_CUDA_PATH, not in the repository rule implementation.

Yes, this is the structure created by the official CUDA installer (run file). The other paths (bin, lib, include, nvvm) exist at the top-level, only CUPTI gets installed to a subdir. So creating symlinks of the CUPTI folders isn't possible and one would need to create symlinks for every CUPTI file.
The repository rule already creates symlinks in cuda_curand, cuda_nvcc, ... (to $LOCAL_CUDA_PATH/include etc.) and if I understood the code for the download correctly it unpacks the CUPTI archive to a new folder and uses that for the cupti* targets/rules
So it seems to make sense to have cuda_cupti symlinks point to $LOCAL_CUDA_PATH/extras/CUPTI which is the root folder of CUPTI and hence mirrors the layout used when the files were downloaded, doesn't it?

What could be an issue with the patch? I guess it would need a check for existence of the extras folder to avoid breaking the current use case of having all in one folder, but besides that?

[vam-google]

tl;dr; I would strictly recommend avoiding LOCAL_CUDA_PATH at all cost. LOCAL_CUDA_PATH exists, but it exist to support a different use case, not yours. Our build already downloads half of the internet to build itself, so downloading cuda stuff, although cuda is one of the heaviest deps we have, it does not make much of a difference (compared to not downloading it), and it should be ok to do it, especially given that it is done only once, and after that it is in bazel cache anyways. If you still want deps to be available locally, another option would be to package them in tar.gz files and pull them with as file:/// iinstead of https://.

There is one topic which I could not find being discussed after "scanning" this thread: the type of Linux distribution you are using.

Why this is important: any discussion about where files end up and/or where official nvidia installed eventually puts them depend on the actual linux distribution you are using, while xla/jax build should not depend on linux distro type.

As you already know the LOCAL_CUDA_PATH thing is not recommended, and in fact, exists solely to cover the use case when the same person develops both: xla/jax and cuda binaries, which is specific to nvidia teams. Everyone else, who does not build custom nvidia binaries should not be using that at all. It does not exist to allow using local preinstalled cuda binaries, that is not the true use case it was added for.

The repository rule already creates symlinks in cuda_curand, cuda_nvcc, ... (to $LOCAL_CUDA_PATH/include etc.) and if I understood the code for the download correctly it unpacks the CUPTI archive to a new folder and uses that for the cupti* targets/rules

The rules create links, but but they do not try to guess the folder structure or to adjust to different linux distributions you may be dealing with. Instead LOCAL_CUDA_PATH defines a specific interface that everyone who wants to use it must adjust to, not the vice versa. I.e. the problem with your patch is the if statements, which conceptually change the logic in those repo rules from "we define folder structure interface we expect" to "we are trying to guess and find stuff on the file system". We need to keep our builds hermetic and reproducible, LOCAL_CUDA_PATH is already breaking that contract, and adding "guessing/finding" files functionality makes it even worse. A simple example why LOCAL_CUDA_PATH is bad: changes in those folders will not be visible to the cache, and you can easily get a broken build due to cache corruption, something we want to avoid at all cost.

[Flamefire]

tl;dr; I would strictly recommend avoiding LOCAL_CUDA_PATH at all cost. LOCAL_CUDA_PATH exists, but it exist to support a different use case, not yours. Our build already downloads half of the internet to build itself, so downloading cuda stuff, although cuda is one of the heaviest deps we have, it does not make much of a difference (compared to not downloading it), and it should be ok to do it, especially given that it is done only once, and after that it is in bazel cache anyways.

For us there is no bazel cache: The build starts always from a clean state so it can be done on any machine that has access to the already installed software modules. What we do "cache" are the downloaded archives of things we do install, especially to allow offline builds as HPC system might not have internet access.
Avoiding the CUDA download would help to cut down on download times as definitely have it available in a defined version although I understand that a lot of other stuff is downloaded anyway, and e.g. LLVM is a big(ger?) chunk.
We make an effort to reduce that as much as possible, e.g. by using TF_SYSTEM_LIBS, and intended to use --experimental_repository_cache, which didn't exist when we started with TF 1.x and we didn't got around using it yet.
But using our installed CUDA worked until recently (~ TF 2.18)

From reading the docs of LOCAL_CUDA_PATH it wasn't clear to me, that this use case isn't supported or even intended.

You can use the local CUDA/CUDNN/NCCL/NVSHMEM dirs as a source of redistributions.

To me this sounds like "local CUDA dir" refers to a local CUDA installation. If that is not the case, as I understand your reply, maybe this can be clarified at that place.

One question I still have is why "LOCAL_CUDA_PATH should be avoided at all cost". For me/us it isn't clear why we can't use what is already installed and will be used to run TF/XLA later which (for virtually all other software) avoids compatibility issues.
Maybe that answer would also be great to have in the above docs as it is an issue that others have run into too. I see they are in the same position as us, just using a different "package/build manager" (for HPC).