Building with CentOS causes libtorch linker errors

[ngrupen]

CentOS 7.8.2003 (Core)
cmake version 3.18.4
clang version 10.0.1
Python 3.8.8

I have successfully built/run OpenSpiel + libtorch on macOS and Ubuntu, but am running into issues when building with CentOS. It is a lab cluster (more compute needed for complex games), so cuda is also installed in a non-standard location. I already fixed a few issues installing libtorch from source because of this (namely, this one) but am now stumped on some linker errors during the torch tests---or really anytime find_package(Torch REQUIRED) is used. I know this is really more of a libtorch issue, but I am curious if anyone else has solved this while building OpenSpiel, because the solution from the libtorch forums seems to be to use a different compiler (see solutions here and here). Of course, clang is required for OpenSpiel, so this is not an option in my case.

Anyways, here is the error:

/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libgomp-75eea7e8.so.1: undefined reference to `clock_gettime@GLIBC_2.17'
/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libgomp-75eea7e8.so.1: undefined reference to `clock_getres@GLIBC_2.17'
/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libc10.so: undefined reference to `memcpy@GLIBC_2.14'
/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libtorch_cpu.so: undefined reference to `lgammaf@GLIBC_2.23'
/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libtorch_cpu.so: undefined reference to `lgamma@GLIBC_2.23'
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [algorithms/dqn_torch/dqn_torch_test] Error 1
make[1]: *** [algorithms/dqn_torch/CMakeFiles/dqn_torch_test.dir/all] Error 2
/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libgomp-75eea7e8.so.1: undefined reference to `clock_gettime@GLIBC_2.17'
/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libgomp-75eea7e8.so.1: undefined reference to `clock_getres@GLIBC_2.17'
/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libc10.so: undefined reference to `memcpy@GLIBC_2.14'
/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libtorch_cpu.so: undefined reference to `lgammaf@GLIBC_2.23'
/home/fs01/nag83/anaconda2/envs/spiel/bin/ld: /home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libtorch_cpu.so: undefined reference to `lgamma@GLIBC_2.23'
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [algorithms/alpha_zero_torch/torch_vpnet_test] Error 1
make[1]: *** [algorithms/alpha_zero_torch/CMakeFiles/torch_vpnet_test.dir/all] Error 2

Any advice is much appreciated!

[lanctot]

Hi @ngrupen,

Of course, clang is required for OpenSpiel, so this is not an option in my case.

Actually, I recently discovered it still works perfectly fine with g++. In fact, we will need to go back to "fully" supporting g++ compilation because it is required for building the binary distribution wheels that we plan to use on the Python side as of the next release (see this PR and some discussion on this thread).

So, we've since toned down the wording about "requiring" clang (to just "recommending" it), but we probably didn't catch all instances. If you know of any place in the docs where it's still listed as required, please let us know.

Hope this helps!

[ngrupen]

That's good to know! I actually didn't see that clang++ was required anywhere, I was just going with what I remembered from the last time I set things up :)

Will give it a shot with g++ and report back.

[lanctot]

Great, let us know how it goes. Tagging @christianjans who might also be interested or have some ideas.

[ngrupen]

Hmm, it also seems to be failing with gcc=9.3 and g++=9.3. I noticed that glibc (2.17) is quite a bit older on the head node of the cluster than on the other machines I've built OpenSpiel on. This might be the cause of the libtorch issues, but looking into this further.

[lanctot]

Ok. Same error?

The functions causing the problems seem too related to this thread and this thread that you mentioned to be a conicidence.

From my reading, it seems related to using the compilers within the conda environment. The fix seems to be: don't do that, use the system's gcc/g++, have you tried that.. e.g. have you tried this suggestion: pytorch/pytorch#27834 (comment) ?

[ngrupen]

Yes I tried those instructions earlier. Setting to the system defaults returns the following errors:

/home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libtorch_cpu.so: undefined reference to `lgammaf@GLIBC_2.23'
/home/fs01/nag83/open_spiel/open_spiel/libtorch/libtorch/lib/libtorch_cpu.so: undefined reference to `lgamma@GLIBC_2.23'

which is an improvement, but the system g++ is 7.3.0 (below the 9.2 mentioned in the installation instructions).

Update: ^^ The above was actually for a different gcc/g++ version that was in a custom location. The real system versions (/usr/bin/gcc, etc.) are even lower (4.8.5), so I'm not sure if it is feasible to use them.

[lanctot]

Ok, nonetheless, I feel this is on the libtorch side. If you download libtorch separately, can you build a test or a simple example that runs it on the machine(s) you're using? If it fails then I think it's best to follow-up with the libtorch team.

If that works, then maybe try compiling libtorch from scratch? (This was one of the suggestions in those threads from above.)

[ngrupen]

The simple example from the libtorch site works on the other machines -- it's actually what I'm using primarily to debug this. Definitely not an OpenSpiel problem, was just curious if anyone here had seen it before. Will close, but will give the other suggestions a shot.

[lanctot]

Ok makes sense. But, it's not a problem to keep open for visibility! Possibly others will run into this and it'd be great if we found a solution here (and maybe can even add to those other threads / issues if we discover more).

[ngrupen]

Ok minor update:
The simple libtorch example builds/runs fine from source. Now going to try pointing OpenSpiel to this version.

Update:
Ta da! Building OpenSpiel with libtorch from source works! As for the reason that it did not build/link with libtorch binaries: ¯_(ツ)_/¯

But I can confirm all of the AlphaZero-related targets linked properly and all the torch-related tests passed. A few minor modifications to the build process to use libtorch from source (building for AlphaZero torch):

• Set OPEN_SPIEL_BUILD_WITH_LIBNOP and OPEN_SPIEL_BUILD_WITH_LIBTORCH, as in AlphaZero instructions.
• In install.sh, comment out lines that download libtorch binaries (lines 138-143 in current version from master).
• Run install.sh
• Set Torch_DIR environment variable to point to libtorch. For me this was: export Torch_DIR=python -c 'import torch;print(torch.utils.cmake_prefix_path)'
• Run build_and_run_tests.sh with same compilation flags that were used to build libtorch.

p.s. A few references

• Installing from source: here and here.
• Minimal example to verify libtorch: here.

[lanctot]

Nice!!! I love the "reason" :)

I wonder if it's worth adding a link to the this thread in the AlphaZero torch instructions, something like "note, if you are building from CentOS or encounter an error with missing symbols, try this" then link to your summary above? Would you be happy to submit a PR for that?

[christianjans]

Thanks for the tag on this issue @lanctot.

I wonder if it's worth adding a link to the this thread in the AlphaZero torch instructions, something like "note, if you are building from CentOS or encounter an error with missing symbols, try this" then link to your summary above?

This would be a great addition! Let us know if you would like to add that to the LibTorch AlphaZero instructions @ngrupen. Glad you figured out a solution!

[ngrupen]

That sounds great! I'll shoot over a PR.

Update:
PR created --> #622 (comment)