Add libcudf manifest for wheel build in devcontainer

[msarahan]

I'm not at all sure that I've done this correctly. This is intended to support local development with a devcontainer for the work in rapidsai/cudf#15483

[trxcllnt]

The depends list in Python is for defining the source-built C++ libs that the Python project needs in order to build. Yeah, it could definitely be named better 😅.

[vyasr]

Before we merge this we should make sure that we properly handle this use case in all the packages. We're going to have to adapt both the CMake and the python code in the C++ wheels just a little bit (discussed with Paul earlier today).

[jameslamb]

I've revived this now that rapidsai/cudf#15483 is just about ready to be merged.

We're going to have to adapt both the CMake and the python code in the C++ wheels just a little bit (discussed with Paul earlier today).

@vyasr could you share the details for this statement? What else needs to be done?

[jameslamb]

@vyasr could you share the details for this statement? What else needs to be done?

We talked about this offline. Summarizing here.

In wheel builds in cudf CI, compilation of libcudf.so happens in the libcudf wheel build. That wheel contains libcudf.so, and libraries that need it (cudf and pylibcudf) call libcudf.load_library() to dlopen it.

Here in devcontainers, libcudf.so is compiled outside of any wheel-building by just directly invoking cmake. The libcudf wheel won't include it, and so libcudf.load_library() will not load it. That's fine... as long as libcudf.so is installed in a location that the linker can find when something in cudf / pylibcudf requires it.

I tested this and it looks like it's working as expected. Rebuilt the CUDA 12.0 pip devcontainer from this branch, with only cudf (from rapidsai/cudf#15483) mounted in.

Ran the following:

# build libcudf.so + all the Python wheels
build-cudf -j

# test that 'libcudf' is importable
python -c "import libcudf; l = libcudf.load_library(); print(l)"
# None

# test that 'pylibcudf' and 'cudf' are importable
python -c "import pylibcudf;import cudf"

# test that cudf actually works
./cudf/ci/run_cudf_pytests.sh

Saw all the expected things:

• libcudf.so was only built once, and all the wheel builds found it without rebuilding it
• libcudf, cudf, and pylibcudf wheels were all built successfully
• libcudf, cudf, and pylibcudf were all importable
• the cudf tests all run successfully

So I think this can be merged, and should be prior to rapidsai/cudf#15483 being merged.

[vyasr]

Thanks for documenting James.