SVD test segfaults on Apple M1

[Keno]

      From worker 6:
      From worker 6:	signal (11): Segmentation fault: 11
      From worker 6:	in expression starting at /Users/keno/julia/usr/share/julia/stdlib/v1.8/LinearAlgebra/test/svd.jl:68
      From worker 6:	ntuple at ./ntuple.jl:0
      From worker 6:	unknown function (ip: 0x115d6f2f3)
      From worker 6:	_jl_invoke at /Users/keno/julia/src/gf.c:0 [inlined]
      From worker 6:	jl_apply_generic at /Users/keno/julia/src/gf.c:2427
      From worker 6:	getindex at ./range.jl:373
      From worker 6:	_hvcat_rows at /Users/keno/julia/usr/share/julia/stdlib/v1.8/SparseArrays/src/sparsevector.jl:1114
      From worker 6:	_jl_invoke at /Users/keno/julia/src/gf.c:0 [inlined]
      From worker 6:	jl_apply_generic at /Users/keno/julia/src/gf.c:2427

[Keno]

Sigh, this is some sort of nasty something. Goes away under LLDB. Also no rr available 😢 .

[giordano]

This might be related: JuliaStats/Distributions.jl#1344

[giordano]

Also @chriselrod found some weird segmentation faults referencing ntuple at ./ntuple.jl:0.

[Keno]

Yes, ntuple seems to be a recurring theme in these.

[giordano]

What I found most fun in JuliaStats/Distributions.jl#1344 is that it isn't deterministic, one more reason to miss rr 🙃

[giordano]

For the record, I just ran

julia stdlib/LinearAlgebra/test/runtests.jl

on latest master and all tests are successful for me. Also julia stdlib/LinearAlgebra/test/svd.jl wouldn't reproduce the issue, but I can still reproduce JuliaStats/Distributions.jl#1344. This bug is really annoying to track down.

---

Edit: I can reproduce the issue by running the tests with

% julia runtests.jl LinearAlgebra
Test                          (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
LinearAlgebra/special              (7) |        started at 2021-07-05T00:19:07.868
LinearAlgebra/dense                (4) |        started at 2021-07-05T00:19:07.903
LinearAlgebra/eigen                (8) |        started at 2021-07-05T00:19:07.954
LinearAlgebra/bunchkaufman         (9) |        started at 2021-07-05T00:19:07.954
LinearAlgebra/qr                   (3) |        started at 2021-07-05T00:19:07.954
LinearAlgebra/triangular           (2) |        started at 2021-07-05T00:19:07.955
LinearAlgebra/schur                (6) |        started at 2021-07-05T00:19:07.955
LinearAlgebra/matmul               (5) |        started at 2021-07-05T00:19:07.956
LinearAlgebra/schur                (6) |    36.94 |   1.18 |  3.2 |    3824.11 |   411.69
LinearAlgebra/svd                  (6) |        started at 2021-07-05T00:19:45.171
LinearAlgebra/bunchkaufman         (9) |    40.64 |   1.84 |  4.5 |    5932.33 |   481.34
LinearAlgebra/lapack               (9) |        started at 2021-07-05T00:19:48.722
      From worker 6:
      From worker 6:	signal (11): Segmentation fault: 11
      From worker 6:	in expression starting at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/LinearAlgebra/test/svd.jl:68
      From worker 6:	ntuple at ./ntuple.jl:0
      From worker 6:	unknown function (ip: 0x1154ddbb3)
      From worker 6:	_jl_invoke at /Users/mose/repo/julia/src/gf.c:0 [inlined]
      From worker 6:	jl_apply_generic at /Users/mose/repo/julia/src/gf.c:2427
      From worker 6:	getindex at ./range.jl:373
      From worker 6:	_hvcat_rows at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/SparseArrays/src/sparsevector.jl:1117
      From worker 6:	_jl_invoke at /Users/mose/repo/julia/src/gf.c:0 [inlined]
      From worker 6:	jl_apply_generic at /Users/mose/repo/julia/src/gf.c:2427
      From worker 6:	jl_apply at /Users/mose/repo/julia/src/./julia.h:1787 [inlined]
      From worker 6:	do_apply at /Users/mose/repo/julia/src/builtins.c:713
      From worker 6:	hvcat at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/SparseArrays/src/sparsevector.jl:1107
      From worker 6:	getproperty at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/svd.jl:482
      From worker 6:	unknown function (ip: 0x1154d928f)
      From worker 6:	_jl_invoke at /Users/mose/repo/julia/src/gf.c:0 [inlined]
      From worker 6:	jl_apply_generic at /Users/mose/repo/julia/src/gf.c:2427
      From worker 6:	macro expansion at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:445 [inlined]
      From worker 6:	macro expansion at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/LinearAlgebra/test/svd.jl:106 [inlined]
      From worker 6:	macro expansion at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1282 [inlined]
      From worker 6:	macro expansion at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/LinearAlgebra/test/svd.jl:103 [inlined]
      From worker 6:	top-level scope at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
      From worker 6:	top-level scope at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/LinearAlgebra/test/svd.jl:0
      From worker 6:	jl_toplevel_eval_flex at /Users/mose/repo/julia/src/toplevel.c:876
      From worker 6:	jl_eval_module_expr at /Users/mose/repo/julia/src/toplevel.c:196 [inlined]
      From worker 6:	jl_toplevel_eval_flex at /Users/mose/repo/julia/src/toplevel.c:673
      From worker 6:	jl_toplevel_eval_flex at /Users/mose/repo/julia/src/toplevel.c:830
      From worker 6:	jl_toplevel_eval at /Users/mose/repo/julia/src/toplevel.c:894 [inlined]
      From worker 6:	jl_toplevel_eval_in at /Users/mose/repo/julia/src/toplevel.c:944
      From worker 6:	eval at ./boot.jl:373 [inlined]
      From worker 6:	include_string at ./loading.jl:1196
      From worker 6:	_jl_invoke at /Users/mose/repo/julia/src/gf.c:0 [inlined]
      From worker 6:	jl_apply_generic at /Users/mose/repo/julia/src/gf.c:2427
      From worker 6:	_include at ./loading.jl:1253
      From worker 6:	include at ./Base.jl:417 [inlined]
      From worker 6:	macro expansion at /Users/mose/repo/julia/test/testdefs.jl:24 [inlined]
      From worker 6:	macro expansion at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1282 [inlined]
      From worker 6:	macro expansion at /Users/mose/repo/julia/test/testdefs.jl:23 [inlined]
      From worker 6:	macro expansion at ./timing.jl:368 [inlined]
      From worker 6:	#runtests#1 at /Users/mose/repo/julia/test/testdefs.jl:21
      From worker 6:	runtests##kw at /Users/mose/repo/julia/test/testdefs.jl:6 [inlined]
      From worker 6:	runtests##kw at /Users/mose/repo/julia/test/testdefs.jl:6
      From worker 6:	unknown function (ip: 0x1152e7fb7)
      From worker 6:	_jl_invoke at /Users/mose/repo/julia/src/gf.c:0 [inlined]
      From worker 6:	jl_apply_generic at /Users/mose/repo/julia/src/gf.c:2427
      From worker 6:	jl_apply at /Users/mose/repo/julia/src/./julia.h:1787 [inlined]
      From worker 6:	do_apply at /Users/mose/repo/julia/src/builtins.c:713
      From worker 6:	#106 at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:278
      From worker 6:	run_work_thunk at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:63
      From worker 6:	macro expansion at /Users/mose/repo/julia/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:278 [inlined]
      From worker 6:	#105 at ./task.jl:411
      From worker 6:	unknown function (ip: 0x1152dc69f)
      From worker 6:	_jl_invoke at /Users/mose/repo/julia/src/gf.c:0 [inlined]
      From worker 6:	jl_apply_generic at /Users/mose/repo/julia/src/gf.c:2427
      From worker 6:	jl_apply at /Users/mose/repo/julia/src/./julia.h:1787 [inlined]
      From worker 6:	start_task at /Users/mose/repo/julia/src/task.c:880
      From worker 6:	Allocations: 118139621 (Pool: 118071927; Big: 67694); GC: 129

which I guess is what Keno did.

[gbaraldi]

Ok, I was running this test and got a kernel panic, I haven't tried to reproduce it yet, the panicked task was WindowServer but I don't know if it was just a coincidence
Edit:
I tried to reproduce and failed. The panic happened while running the tests like this : julia stdlib/LinearAlgebra/test/runtests.jl. I tried running it that way again and it just worked. Running the test the way Mose showed also passed normally.

could this be related?

dmlc/xgboost#7039

[giordano]

I don't think so, since we we don't use LLVM libomp anywhere.

[gbaraldi]

Hvcat seems to be a common denominator https://gist.github.com/gbaraldi/56f2d34fe841a182d6f29a0078830f83
I was testing a PR and got 5 ntuple errors, with 4 being related to hvcat.