Skip to content

Conversation

@Flamefire
Copy link
Contributor

Currently we used /tmp (or better: tempdir as set by EB options) to put bazel build files in. However often users set builddir to some fast storage (e.g. /dev/shm) so using that instead can speed up the build of TensorFlow. This PR solves that and also unifies some settings of various directories used

Additionally it removes some patches:

  • The configure.py patch doesn't apply since 1.12.0: tensorflow/tensorflow@56a99d6 As this hasn't caused any issues yet it can be removed. It tried building some TF 2.x and 1.15 versions and no .cache/bazel folder is created
  • Also I was told that TF is much more friendly to new toolchains. I hence tested removing our various, extensive patching of CROSSTOOL files with some TF 2.x ECs and it seems to be indeed no longer required. Bazel/TF seem to detect the built-in directories just well, so no need to manually specify them. I don't know when that happened (and am unsure how to find that) so I simply used 2.0 as the cutoff point. We've got 1.13.1 for 2019a, so I could test and use that as well, if wanted. Factoring this out into a separate function greatly simplifies the code making it more readable.

@Flamefire
Copy link
Contributor Author

Flamefire commented Sep 10, 2020

Test:

x86

  • TensorFlow-1.13.1-foss-2019a-Python-3.7.2.eb
  • TensorFlow-1.13.1-fosscuda-2019a-Python-3.7.2.eb
  • TensorFlow-1.14.0-foss-2019a-Python-3.7.2.eb
  • TensorFlow-1.14.0-fosscuda-2019a-Python-3.7.2.eb
  • TensorFlow-1.15.0-foss-2019b-Python-3.7.4.eb
  • TensorFlow-1.15.0-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-1.15.2-foss-2019b-Python-3.7.4.eb
  • TensorFlow-1.15.2-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-2.0.0-foss-2019a-Python-3.7.2.eb
  • TensorFlow-2.0.0-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-2.0.1-foss-2019a-Python-3.7.2.eb
  • TensorFlow-2.0.1-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb
  • TensorFlow-2.1.0-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-2.2.0-foss-2019b-Python-3.7.4.eb
  • TensorFlow-2.2.0-fosscuda-2019b-Python-3.7.4.eb

POWER

  • TensorFlow-1.15.0-foss-2019b-Python-3.7.4.eb
  • TensorFlow-1.15.0-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-1.15.2-foss-2019b-Python-3.7.4.eb
  • TensorFlow-1.15.2-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-2.0.0-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-2.0.1-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-2.1.0-foss-2019b-Python-3.7.4.eb
  • TensorFlow-2.1.0-fosscuda-2019b-Python-3.7.4.eb
  • TensorFlow-2.2.0-foss-2019b-Python-3.7.4.eb
  • TensorFlow-2.2.0-fosscuda-2019b-Python-3.7.4.eb

@Flamefire
Copy link
Contributor Author

@boegel All tests finished successfully. The 1.13.1 fails due to easybuilders/easybuild-easyconfigs#11253

So IMO good to go.

Co-authored-by: Kenneth Hoste <[email protected]>
@boegel boegel changed the title [TensorFlow] Remove some patches and put build files into builddir update TensorFlow easyblock to put Bazel build files in build directory + avoid unnecessary runtime patching Sep 24, 2020
@boegel
Copy link
Member

boegel commented Sep 24, 2020

I've also tested this with a wide range of TensorFlow easyconfigs, no problems encountered whatsoever, so good to go.

Thanks a lot for your efforts on this @Flamefire!

@boegel boegel merged commit 68971a3 into easybuilders:develop Sep 24, 2020
@Flamefire Flamefire deleted the tf_improvements branch September 24, 2020 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants