fix numpy tests for recent SciPy-bundle easyconfig on POWER #12481

Flamefire · 2021-03-29T10:54:42Z

(created using eb --new-pr)

verdurin · 2021-03-29T16:12:19Z

Test build works for me with SciPy-bundle-2019.10-fosscuda-2019b-Python-3.7.4.eb, just not able to upload a report from that node currently.

Flamefire · 2021-03-29T21:15:07Z

Test report by @Flamefire
FAILED
Build succeeded for 9 out of 11 (11 easyconfigs in total)
taurusa6 - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), Python 2.7.5
See https://gist.github.com/639cd1aa179edf0d04756c9a3d35c01e for a full test report.

Flamefire · 2021-03-29T22:45:49Z

Test report by @Flamefire
FAILED
Build succeeded for 16 out of 28 (11 easyconfigs in total)
taurusml24 - Linux RHEL 7.6, POWER, 8335-GTX (power9le), Python 2.7.5
See https://gist.github.com/642ab70a106eaa87b2e384f12d28db13 for a full test report.

smoors · 2021-03-30T13:16:54Z

@boegelbot: please test @ generoso

boegelbot · 2021-03-30T13:20:08Z

@smoors: Request for testing this PR well received on generoso

PR test command 'EB_PR=12481 EB_ARGS= /apps/slurm/default/bin/sbatch --job-name test_PR_12481 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 16457

Test results coming soon (I hope)...

- notification for comment with ID 810230480 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegel

lgtm

Flamefire · 2021-03-30T13:24:32Z

FYI: Tests are being rerun now after fixing the patch and getting GDRcopy installed via the CUDA easyblock change. There might be a semi-expected failure on POWER and 2019a due to know issues with OpenBLAS, but lets see

boegel · 2021-03-30T16:41:21Z

Test reports from CentOS 7 (Haswell) and RHEL8 (Rome) coming up from my side too...

Flamefire · 2021-03-30T16:52:12Z

Test report by @Flamefire
FAILED
Build succeeded for 8 out of 11 (11 easyconfigs in total)
taurusml6 - Linux RHEL 7.6, POWER, 8335-GTX (power9le), Python 2.7.5
See https://gist.github.com/c86966bf73a709420aac13476429dedd for a full test report.

boegelbot · 2021-03-30T19:11:30Z

Test report by @boegelbot
FAILED
Build succeeded for 10 out of 11 (11 easyconfigs in total)
generoso-c1-s-1 - Linux centos linux 8.2.2004, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/f2cf2350ff34ce1594bb9ef09507cf6f for a full test report.

Flamefire · 2021-03-30T21:25:46Z

Test report by @Flamefire
SUCCESS
Build succeeded for 11 out of 11 (11 easyconfigs in total)
taurusa5 - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), Python 2.7.5
See https://gist.github.com/0b3b12941b115a36bf008e507cc97a9a for a full test report.

boegel · 2021-03-30T21:39:46Z

Test report by @boegel
FAILED
Build succeeded for 10 out of 11 (11 easyconfigs in total)
node3501.doduo.os - Linux RHEL 8.2, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/80878612c4a7c57455de42a4e2cef362 for a full test report.

boegel · 2021-03-30T23:54:43Z

Test report by @boegel
SUCCESS
Build succeeded for 11 out of 11 (11 easyconfigs in total)
node2609.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/7318a5e375377de355708dd7d99c9b6b for a full test report.

boegel · 2021-03-31T08:30:17Z

@boegelbot please test @ generoso
EB_ARGS="SciPy-bundle-2019.10-fosscuda-2019b-Python-2.7.16.eb"

Flamefire · 2021-03-31T08:31:52Z

@boegel Your CUDA/10.1.105-GCC-8.2.0-2.31.1 might be broken.
As mentioned, the 2019a failures are kinda expected. I'd even vote to bump the OpenBLAS version for PPC for that TC
The fail with Python 2 2020a is related to numpy/numpy#7601 (comment) where gcc with -O1 -mcpu=power9 miscompiles a loop
Edit: Our ECs have a patch for that issue. So this can be merged from my side (not rerunning the lengthy test for this)

boegelbot · 2021-03-31T08:35:06Z

@boegel: Request for testing this PR well received on generoso

PR test command 'EB_PR=12481 EB_ARGS="SciPy-bundle-2019.10-fosscuda-2019b-Python-2.7.16.eb" /apps/slurm/default/bin/sbatch --job-name test_PR_12481 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 16470

Test results coming soon (I hope)...

- notification for comment with ID 810882672 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2021-03-31T09:00:10Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
generoso-c1-s-1 - Linux centos linux 8.2.2004, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/915171591eb1df23ce6addb50ff5d363 for a full test report.

branfosj · 2021-03-31T12:45:35Z

As mentioned, the 2019a failures are kinda expected. I'd even vote to bump the OpenBLAS version for PPC for that TC
The fail with Python 2 2020a is related to numpy/numpy#7601 (comment) where gcc with -O1 -mcpu=power9 miscompiles a loop
Edit: Our ECs have a patch for that issue. So this can be merged from my side (not rerunning the lengthy test for this)

We bumped our OpenBLAS version for 2019a. I think it would be sensible to do so in the main EB repo as well - though I'd be fine with that being a separate PR if we are all in agreement with doing it.

I've set off test reports from our P9 of the 2019b, 2020a, and 2020b Python 3 versions.

branfosj · 2021-03-31T14:12:36Z

Test report by @branfosj
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
bear-pg0306u07a.bear.cluster - Linux RHEL 8.3, POWER, 8335-GTX (power9le), Python 3.6.8
See https://gist.github.com/214223a13bd2ad1d620f02c87662ccba for a full test report.

branfosj · 2021-03-31T14:25:27Z

Test report by @branfosj
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
bear-pg0306u07a.bear.cluster - Linux RHEL 8.3, POWER, 8335-GTX (power9le), Python 3.6.8
See https://gist.github.com/913cfdc3c5214c5c97343c2713f0bf68 for a full test report.

branfosj · 2021-03-31T14:25:57Z

Test report by @branfosj
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
bear-pg0306u07a.bear.cluster - Linux RHEL 8.3, POWER, 8335-GTX (power9le), Python 3.6.8
See https://gist.github.com/13da25a0c1e6b60516b5512bf697ab65 for a full test report.

boegel · 2021-04-01T16:30:40Z

Going in, thanks @Flamefire!

[SciPy-bundle] Fix numpy tests on PPC

18c9fe6

This comment has been minimized.

Sign in to view

Flamefire added 2 commits March 29, 2021 13:28

Enable pip check

6474cb0

Use https

5530034

Update patches for changes in numpy 1.19

b015cb3

boegel approved these changes Mar 30, 2021

View reviewed changes

boegel added the bug fix label Mar 30, 2021

boegel added this to the next release (4.3.4?) milestone Mar 30, 2021

boegel changed the title ~~[SciPy-bundle] Fix numpy tests on PPC~~ fix numpy tests for recent SciPy-bundle easyconfig on POWER Mar 31, 2021

boegel merged commit 504844a into easybuilders:develop Apr 1, 2021

Flamefire deleted the 20210329125435_new_pr_SciPy-bundle201903 branch April 1, 2021 17:54

fix numpy tests for recent SciPy-bundle easyconfig on POWER #12481

fix numpy tests for recent SciPy-bundle easyconfig on POWER #12481

Uh oh!

Conversation

Flamefire commented Mar 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

verdurin commented Mar 29, 2021

Uh oh!

Flamefire commented Mar 29, 2021

Uh oh!

Flamefire commented Mar 29, 2021

Uh oh!

smoors commented Mar 30, 2021

Uh oh!

boegelbot commented Mar 30, 2021

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

Flamefire commented Mar 30, 2021

Uh oh!

boegel commented Mar 30, 2021

Uh oh!

Flamefire commented Mar 30, 2021

Uh oh!

boegelbot commented Mar 30, 2021

Uh oh!

Flamefire commented Mar 30, 2021

Uh oh!

boegel commented Mar 30, 2021

Uh oh!

boegel commented Mar 30, 2021

Uh oh!

boegel commented Mar 31, 2021

Uh oh!

Flamefire commented Mar 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegelbot commented Mar 31, 2021

Uh oh!

boegelbot commented Mar 31, 2021

Uh oh!

branfosj commented Mar 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

branfosj commented Mar 31, 2021

Uh oh!

branfosj commented Mar 31, 2021

Uh oh!

branfosj commented Mar 31, 2021

Uh oh!

boegel commented Apr 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Flamefire commented Mar 29, 2021 •

edited

Loading

Flamefire commented Mar 31, 2021 •

edited

Loading

branfosj commented Mar 31, 2021 •

edited

Loading