in-place update of TensorFlow 2.3.0 easyconfigs to version 2.3.1 #11375

Flamefire · 2020-09-25T11:32:41Z

Contains security fixes
Removes the superflous keras-applications package and scipy package

edit (@boegel): OK because TensorFlow 2.3.0 easyconfigs have only been merged very recently into develop via #11040...

Contains security fixes Removes the superflous keras-applications package and scipy package

Flamefire · 2020-09-25T13:13:19Z

Test report by @Flamefire
FAILED
Build succeeded for 2 out of 3 (3 easyconfigs in this PR)
taurusml3 - Linux RHEL 7.6, POWER, 8335-GTX, Python 2.7.5
See https://gist.github.com/cfd01771b6106380b7f6f37875fdcc44 for a full test report.

Flamefire · 2020-09-25T13:39:26Z

Test report by @Flamefire
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in this PR)
taurusi5196.taurus.hrsk.tu-dresden.de - Linux RHEL 7.8, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/3ac6fe27d1d954b0ad727964feffd717 for a full test report.

boegel · 2020-09-25T16:00:45Z

@Flamefire Again trouble on POWER?

Flamefire · 2020-09-25T16:02:10Z

Yes -.- Although it looks like a flake in the filesystem. Restarted the build but it's been running for 2 hrs now

lexming · 2020-09-25T16:02:47Z

Test report by @lexming
FAILED
Build succeeded for 6 out of 9 (3 easyconfigs in this PR)
node375.hydra.os - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, Python 2.7.5
See https://gist.github.com/6774d91ddf80b76e8885b23fe77443bf for a full test report.

boegel · 2020-09-25T16:02:56Z

@boegelbot please test @ generoso

boegelbot · 2020-09-25T16:05:09Z

@boegel: Request for testing this PR well received on generoso

PR test command 'EB_PR=11375 EB_ARGS= /apps/slurm/default/bin/sbatch --job-name test_PR_11375 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 7966

Test results coming soon (I hope)...

- notification for comment with ID 699013665 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

Flamefire · 2020-09-25T16:05:19Z

@lexming please rebuild double-conversion

Flamefire · 2020-09-25T16:24:19Z

Test report by @Flamefire
FAILED
Build succeeded for 2 out of 3 (3 easyconfigs in this PR)
taurusml21 - Linux RHEL 7.6, POWER, 8335-GTX, Python 2.7.5
See https://gist.github.com/ea1fa002035b7dee585a72be9bbb4986 for a full test report.

boegel · 2020-09-25T16:49:43Z

Test report by @boegel
FAILED
Build succeeded for 9 out of 12 (3 easyconfigs in this PR)
node3162.skitty.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 3.6.8
See https://gist.github.com/f721507b109546e0009afd0e44fac959 for a full test report.

boegel · 2020-09-25T17:15:50Z

Test report by @boegel
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in this PR)
node3407.kirlia.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz (cascadelake), Python 2.7.5
See https://gist.github.com/1baa102fa06b0afc4d985b4c6148334e for a full test report.

boegel · 2020-09-25T17:52:40Z

Test report by @boegel
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in this PR)
node2706.swalot.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 2.7.5
See https://gist.github.com/d57da72dce96f15e3ee67aef7c12a3d2 for a full test report.

boegelbot · 2020-09-25T18:19:29Z

Test report by @boegelbot
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in this PR)
generoso-x-2 - Linux centos linux 8.2.2004, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/209154df9926efb4b21a79815ad726d6 for a full test report.

lexming · 2020-09-26T22:20:13Z

Test report by @lexming
FAILED
Build succeeded for 1 out of 3 (3 easyconfigs in this PR)
node157.hydra.os - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, Python 2.7.5
See https://gist.github.com/8838d994f39f4eb2a319fb642b47306d for a full test report.

Flamefire · 2020-09-28T08:59:44Z

@lexming

ModuleNotFoundError: No module named 'google.protobuf'

That is strange. You can try loading the build environment and check that this works? If it doesn't there is likely another google folder in your PYTHONPATH(=/user/brussel/101/vsc10122/.local/easybuild-ivybridge/software/TensorFlow/2.3.1-foss-2019b-Python-3.7.4/lib/python3.7/site-packages:/user/brussel/101/vsc10122/.local/easybuild-ivybridge/software/TensorFlow/2.3.1-foss-2019b-Python-3.7.4/lib/python3.7/site-packages:/apps/brussel/CO7/ivybridge-ib/software/protobuf-python/3.10.0-foss-2019b-Python-3.7.4/lib/python3.7/site-packages:/apps/brussel/CO7/ivybridge-ib/software/h5py/2.10.0-foss-2019b-Python-3.7.4/lib/python3.7/site-packages:/apps/brussel/CO7/ivybridge-ib/software/SciPy-bundle/2019.10-foss-2019b-Python-3.7.4/lib/python3.7/site-packages:/apps/brussel/CO7/ivybridge-ib/software/Python/3.7.4-GCCcore-8.3.0/easybuild/python)

Check each of those for a google/__init__.py which is what mostly prevents finding other google packages

boegel · 2020-09-30T06:53:09Z

@Flamefire Is this related to #11143?

@lexming Try re-installing protobuf-python-3.10.0-fosscuda-2019b-Python-3.7.4.eb and protobuf-python-3.10.0-foss-2019b-Python-3.7.4.eb with latest develop branch or EasyBuild v4.3.0?

Flamefire · 2020-09-30T07:35:32Z

Ah, yes exactly. That was the reason for the breaking change with the downloaded archive

lexming · 2020-09-30T20:55:27Z

@Flamefire @boegel thanks for the feedback, testing again

lexming

Horovod fails to detect NCCL in our system

     -- Linking against static NCCL library
     CMake Error at /theia/home/apps/CO7/skylake/software/CMake/3.15.3-GCCcore-8.3.0/share/cmake-3.15/Modules/FindPackageHandleStandardArgs    .cmake:137 (message):
       Could NOT find NCCL (missing: NCCL_INCLUDE_DIR NCCL_LIBRARY)
     Call Stack (most recent call first):
       /theia/home/apps/CO7/skylake/software/CMake/3.15.3-GCCcore-8.3.0/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:378 (_    FPHSA_FAILURE_MESSAGE)
       cmake/Modules/FindNCCL.cmake:42 (find_package_handle_standard_args)
       CMakeLists.txt:174 (find_package)
  
     
     -- Configuring incomplete, errors occurred!

I fixed it by adding pkg-config as a build dependency of Horovod

Flamefire · 2020-10-01T12:38:31Z

Hm, that doesn't make sense, that module doesn't use pkg-config at all so adding it shouldn't change anything.

Can you check ml show NCCL/2.4.8-gcccuda-2019b that it contains CMAKE_PREFIX_PATH? Can you also check what loading pkg-config changes? Maybe append the output of >ml show on that and from the succeeding build check the Found NCCL line and paste that (should have a location)

PS: Just seen that there have been 2 bugfix releases to Horovod by now. @boegel Can I updated that too in this PR or a follow up? In particular it includes horovod/horovod#2272 which seems to make certain use cases working again, otherwise an import of a horovod file will fail

lexming · 2020-10-01T20:58:15Z

Test report by @lexming
FAILED
Build succeeded for 2 out of 3 (3 easyconfigs in this PR)
node375.hydra.os - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, Python 2.7.5
See https://gist.github.com/a28d152c7d3eee7e39f9a28d9ec07956 for a full test report.

lexming · 2020-10-01T22:10:03Z

Test report by @lexming
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in this PR)
node101.hydra.os - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, Python 2.7.5
See https://gist.github.com/95b7aa64e77dc9b1d0f9174f0a276856 for a full test report.

boegel · 2020-10-02T07:57:50Z

@Flamefire I would update Horovod in a follow-up PR, let's try and get this one merged first...

boegel · 2020-10-02T07:59:05Z

@lexming The failing test report on node375 is due to a missing $CMAKE_PREFIX_PATH update in the NCCL module?

lexming · 2020-10-02T09:31:09Z

@Flamefire @boegel the real issue is indeed the lack of $CMAKE_PREFIX_PATH in our deployment of NCCL. It should be completely fixed now. Submitting one final test to counter the failed test.

lexming · 2020-10-02T15:41:55Z

Test report by @lexming
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in this PR)
node358.hydra.os - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, Python 2.7.5
See https://gist.github.com/8a0c0dd41bbccbc594c348771d10bc71 for a full test report.

lexming

LGTM

lexming · 2020-10-02T17:11:59Z

Going in, thanks @Flamefire !

Update TensorFlow 2.3.1 and it's dependencies

55e2a7b

Contains security fixes Removes the superflous keras-applications package and scipy package

Flamefire mentioned this pull request Sep 25, 2020

{lib}[fosscuda/2019b] TensorFlow v2.3.0 w/ Python 3.7.4 #11040

Merged

2 tasks

boegel changed the title ~~Update TensorFlow 2.3.1 and it's dependencies~~ in-place update of TensorFlow 2.3.0 easyconfigs to version 2.3.1 Sep 25, 2020

boegel added the update label Sep 25, 2020

boegel added this to the next release (4.3.1) milestone Sep 25, 2020

lexming requested changes Oct 1, 2020

View reviewed changes

lexming approved these changes Oct 2, 2020

View reviewed changes

lexming merged commit 8a83d16 into easybuilders:develop Oct 2, 2020

Flamefire deleted the tf231 branch October 5, 2020 13:39

Flamefire mentioned this pull request Oct 7, 2020

in-place update of easyconfig Horovod v0.20.0 to v0.20.3 #11442

Merged

in-place update of TensorFlow 2.3.0 easyconfigs to version 2.3.1 #11375

in-place update of TensorFlow 2.3.0 easyconfigs to version 2.3.1 #11375

Uh oh!

Conversation

Flamefire commented Sep 25, 2020 • edited by boegel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Flamefire commented Sep 25, 2020

Uh oh!

Flamefire commented Sep 25, 2020

Uh oh!

boegel commented Sep 25, 2020

Uh oh!

Flamefire commented Sep 25, 2020

Uh oh!

lexming commented Sep 25, 2020

Uh oh!

boegel commented Sep 25, 2020

Uh oh!

boegelbot commented Sep 25, 2020

Uh oh!

Flamefire commented Sep 25, 2020

Uh oh!

Flamefire commented Sep 25, 2020

Uh oh!

boegel commented Sep 25, 2020

Uh oh!

boegel commented Sep 25, 2020

Uh oh!

boegel commented Sep 25, 2020

Uh oh!

boegelbot commented Sep 25, 2020

Uh oh!

lexming commented Sep 26, 2020

Uh oh!

Flamefire commented Sep 28, 2020

Uh oh!

boegel commented Sep 30, 2020

Uh oh!

Flamefire commented Sep 30, 2020

Uh oh!

lexming commented Sep 30, 2020

Uh oh!

lexming left a comment

Choose a reason for hiding this comment

Uh oh!

Flamefire commented Oct 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lexming commented Oct 1, 2020

Uh oh!

lexming commented Oct 1, 2020

Uh oh!

boegel commented Oct 2, 2020

Uh oh!

boegel commented Oct 2, 2020

Uh oh!

lexming commented Oct 2, 2020

Uh oh!

lexming commented Oct 2, 2020

Uh oh!

lexming left a comment

Choose a reason for hiding this comment

Uh oh!

lexming commented Oct 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Flamefire commented Sep 25, 2020 •

edited by boegel

Loading

Flamefire commented Oct 1, 2020 •

edited

Loading