Add support for including PTX code in PyTorch

[Flamefire]

This adds PTX code to PyTorch by default for any newer architecture than the last selected one.
This can be changed by the new EC option "ptx"

• Discussion about the cuda cache needs resolving (see below)

QUESTION: What about cuda_cache_size? It might be better to make this an easybuild option (similar to --cuda-compute-capabilities) instead. E.g. for PyTorch the cache seems to be quite large. Running the test test_cpp_extensions_aot_no_ninja alone fills up 1GB

Framework PR: easybuilders/easybuild-framework#3569 If that is merged I can remove the option in this EC

[Flamefire]

Had to add a bugfix as TORCH_CUDA_ARCH_LIST needs to be set for tests too or the build (even current one) will fail if the GPUs found during build are newer than what the used nvcc supports. See https://gist.github.com/3ce737772ff805683c226e500b525c67

[Flamefire]

Test report by @Flamefire

Overview of tested easyconfigs (in order)

• FAIL (build issue) PyTorch-1.7.1-fosscuda-2019b-Python-3.7.4.eb (partial log available at https://gist.github.com/58bf303fda8006243a91887dad037a27)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
taurusi8028 - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7352 24-Core Processor, Python 2.7.5
See https://gist.github.com/d30c8f48f9c0948ea0c543797ddff458 for a full test report.

[branfosj]

The CUDA arches are not guaranteed to be in order, so one of the following changes is required:

1. latest to become last
2. The code below changes to add +PTX to the latest CUDA arch
3. We order the CUDA arches

[Flamefire]

Good catch! Using "last" which matches "first" and is easiest.

[Flamefire]

I had to add an option for setting up the cuda cache as using ptx (by default on) will now possibly trigger JIT compilation which writes to the HOME directory. See https://developer.nvidia.com/blog/cuda-pro-tip-understand-fat-binaries-jit-caching/ for details

There also seems to be a failure which I'm not sure about. Might be because I started the test to early. Will rerun

[Flamefire]

Test report by @Flamefire

Overview of tested easyconfigs (in order)

• SUCCESS PyTorch-1.7.1-fosscuda-2019b-Python-3.7.4.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
taurusml3 - Linux RHEL 7.6, POWER, 8335-GTX, Python 2.7.5
See https://gist.github.com/a2fd3c5a7ab9956d83e1df29e62cb6f7 for a full test report.

[Flamefire]

Test report by @Flamefire

Overview of tested easyconfigs (in order)

• FAIL (build issue) PyTorch-1.7.1-fosscuda-2019b-Python-3.7.4.eb (partial log available at https://gist.github.com/6db857d5e8672a80907e9fd81902f1b0)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
taurusi8033 - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7352 24-Core Processor, Python 2.7.5
See https://gist.github.com/333e42009f7c32b178b429f547deae8b for a full test report.

[Flamefire]

While this does work the amount of JIT compiling that would happen when running the so-compiled PyTorch make this unfeasible. So I'd recommend against using this and am closing it.