BLAS test failure on arm64

[ckastner]

Running test-backend-ops -b BLAS, I'm seeing the following single test failure using the BLAS backend:

MUL_MAT(type_a=f32,type_b=f32,m=16,n=32,k=32,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=64,o=3): not supported [BLAS] [MUL_MAT] ERR = 0.194957976 > 0.000500000 MUL_MAT(type_a=f32,type_b=f32,m=64,n=77,k=77,bs=[12,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): FAIL

The failure is reproducible with different BLAS implementations (on Debian and Ubuntu, these are all interchangeable).

It seems that almost all test-backend-ops tests are skipped with not supported [BLAS] and I also found #293 with the intent to remove BLAS entirely, so perhaps I ran into something obsolete.

[ggerganov]

Which BLAS implementation are you using?

I am not able to reproduce neither on Ubuntu 24.04 with /usr/lib/x86_64-linux-gnu/libblas.so (I think this is OpenBLAS) nor on macOS with Apple's BLAS.

[ckastner]

Huh, I was so focused on the development releases that I didn't think to test the stable ones.

It is indeed also not reproducible for me on 24.04, but it is reproducible on 26.04, and on Debian unstable. Could be a BLAS regression but I can reproduce this with netlib and OpenBLAS, so perhaps this is somewhere lower in the stack?

From what I see, only arm64 is affected. Right now I reproduced on an Orion O6 but earlier tests failed the same way on AWS Graviton4. Both are ARMv9.

Dockerfile for repro: Dockerfile.ggml.1427.txt

$ podman build -t repro-1427 --build-arg UBUNTU_CODENAME=noble . && podman run --rm -it repro-1427
<snip>
8/8 tests passed
  Backend BLAS: OK
Backend 2/2: CPU
  Skipping
2/2 backends passed

$ podman build -t repro-1427 --build-arg UBUNTU_CODENAME=resolute . && podman run --rm -it repro-1427
<snip>
7/8 tests passed
 
Failing tests:
  MUL_MAT(type_a=f32,type_b=f32,m=64,n=77,k=77,bs=[12,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1)
  Backend BLAS: FAIL
Backend 2/2: CPU
  Skipping
1/2 backends passed

[ggerganov]

If the issue is not reproducible with earlier ggml versions, it would indicate something wrong in ggml. In case you can pinpoint a commit after which this starts happening it would be very useful.

[ckastner]

The ggml version I tested was always the same (git HEAD), it's going from 24.04 (good) -> 26.04 (bad) where the failure starts occurring. So technically, it might not even be a ggml problem.

I'll do a package update bisect when I get the chance, that should pinpoint which dependency update is triggering the breakage.

Once we know that, it should be easier to tell if this is a ggml issue, or a BLAS (or other library) issue, or both.