Build 8070 regression: prefill ~75 tok/s instead of ~250 tok/s

[vincentsider]

rocm7-nightlies build 8070 (cc45f2ada) has a severe prefill regression.

Setup:

• GMKtec EVO-X2 (Ryzen AI Max+ 395, 128GB)
• Kernel 6.18.7-200.fc43, firmware 20260110
• Model: Qwen3-Coder-30B-A3B-Instruct BF16

Expected (yesterday): ~250 tok/s prefill
Actual (build 8070): ~75 tok/s prefill
Generation speed: 27 tok/s (correct)

Tested with both ROCBLAS_USE_HIPBLASLT=1 and =0, same result.

Command:
llama-server -m model.gguf --jinja --no-mmap -fa 1 -ngl 999 -c 65536

Image digest: sha256:6c1fc630d304c02c9c726a17407923f9951ac8e18e3e8b84f0c1685d0c15c2ba

[kyuz0]

Thank you for reporting this, are you noticing it only on the rocm7-nightlies or also on 7.2? I will re-run some benchmarks tomorrow and in case this is not fixed, I will open an issue on the ROCm repository.

[vincentsider]

yes i noticed on 7.2.

…

On Mon, 16 Feb 2026 at 21:33, kyuz0 ***@***.***> wrote: *kyuz0* left a comment (kyuz0/amd-strix-halo-toolboxes#58) <#58 (comment)> Thank you for reporting this, are you noticing it only on the rocm7-nightlies or also on 7.2? I will re-run some benchmarks tomorrow and in case this is not fixed, I will open an issue on the ROCm repository. — Reply to this email directly, view it on GitHub <#58 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALZRDBZLPSMEAOFP2IMLMF34MISRJAVCNFSM6AAAAACVKAGJIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMJQGQYTANJYGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[vincentsider]

have you archived builds from late January/early February (31st Jan or 1st Feb), that would be super useful! many thanks for your help

…

On Mon, 16 Feb 2026 at 21:33, kyuz0 ***@***.***> wrote: *kyuz0* left a comment (kyuz0/amd-strix-halo-toolboxes#58) <#58 (comment)> Thank you for reporting this, are you noticing it only on the rocm7-nightlies or also on 7.2? I will re-run some benchmarks tomorrow and in case this is not fixed, I will open an issue on the ROCm repository. — Reply to this email directly, view it on GitHub <#58 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALZRDBZLPSMEAOFP2IMLMF34MISRJAVCNFSM6AAAAACVKAGJIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMJQGQYTANJYGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[kyuz0]

have you archived builds from late January/early February (31st Jan or 1st Feb), that would be super useful! many thanks for your help

Unfortunately I have a script that vacuums builds and only keeps the last 3 ones. I will need to change the logic to keep some monthly snapshots at least.

[jwefers]

can confirm, still present. i also rebuilt today (modified with autoparser branch to fix opencode tool calling issues). PP on qwen3-coder-next went from ~400 to ~200

[kyuz0]

can confirm, still present. i also rebuilt today (modified with autoparser branch to fix opencode tool calling issues). PP on qwen3-coder-next went from ~400 to ~200

Just checking, you are also using the cli argument at build time that was recommended to mitigate this and you're still seeing the issue?

[jwefers]

yes i only replaced the ADD for llama.cpp with a COPY of locally cloned llama.cpp where i keep upstream master + autoparser merged. (rocm7-nightlies docker file).

the rest, including build is still yours.

however, i am still in ubuntu 25.10, meaning kernel 6.17.0-14. dumb choice, will set up the machine with opensuse Tumbleweed in the next days.

but there were a brief few days between this rocm pp regression fixed (yay 400 tps) and day before yesterday, where it went to shit again :D

[kyuz0]

however, i am still in ubuntu 25.10, meaning kernel 6.17.0-14. dumb choice, will set up the machine with opensuse Tumbleweed in the next days.

I have never tried OpenSuse with ROCm, but last time I tried it, it was not something I'd feel to recommend to a fellow human being.

but there were a brief few days between this rocm pp regression fixed (yay 400 tps) and day before yesterday, where it went to shit again :D

Argh... I wonder whether who's the culprit: ROCm vs llvm vs llama.cpp kernels. This easily gives me a headache!

[jwefers]

dont worry, i fully assume it goes away eventually in some newer version.

i didnt think about rocm install on opensuse, but i assume i would only need amdgpu-dkms, since rocm would be inside the container

[vincentsider]

@jwefers : how did you get Qwen3-Coder-Next + llama.cpp on Strix Halo (gfx1151) I thought there 's an unresolved crash bug (GitHub issues #19304, #19355, #19421) affecting Qwen3-Coder-Next specifically on Strix Halo (AMD Ryzen AI MAX+ 395 / gfx1151) , ROCm backends, Long context + tool calling scenarios The crash is: std::runtime_error: Unexpected empty grammar stack after accepting piece This happens regardless of llama.cpp build version , people have reported it on builds from 7925 to 7951+. It's a model + backend interaction bug, not something fixed yet or has it ? How did you do ?

…

On Fri, 20 Feb 2026 at 16:16, jwefers ***@***.***> wrote: *jwefers* left a comment (kyuz0/amd-strix-halo-toolboxes#58) <#58 (comment)> can confirm, still present. i also rebuilt today (modified with autoparser branch to fix opencode tool calling issues). PP on qwen3-coder-next went from ~400 back to ~200 — Reply to this email directly, view it on GitHub <#58 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALZRDB2YDQFCJDKMYNEPDMD4M4QMHAVCNFSM6AAAAACVKAGJIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMZVGUZTKMZSHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[vincentsider]

Kyuz0, my apologies, i did not notice the cli argument at build time, can you point to the resource ?

…

On Fri, 20 Feb 2026 at 16:33, kyuz0 ***@***.***> wrote: *kyuz0* left a comment (kyuz0/amd-strix-halo-toolboxes#58) <#58 (comment)> can confirm, still present. i also rebuilt today (modified with autoparser branch to fix opencode tool calling issues). PP on qwen3-coder-next went from ~400 to ~200 Just checking, you are also using the cli argument at build time that was recommended to mitigate this and you're still seeing the issue? — Reply to this email directly, view it on GitHub <#58 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALZRDBZ7OXXTWF22QR6P2HT4M4SLZAVCNFSM6AAAAACVKAGJIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMZVGYYTOMRUGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jwefers]

@vincentsider by using llama.cpp with the pwilkins autoparser branch merged. i can drop you the instructions in a few hours

[vincentsider]

that would be lovely, thanks a million!

…

On Fri, 20 Feb 2026 at 16:54, jwefers ***@***.***> wrote: *jwefers* left a comment (kyuz0/amd-strix-halo-toolboxes#58) <#58 (comment)> @vincentsider <https://github.com/vincentsider> by using llama.cpp with the pwilkins autoparser branch merged. i can drop you the instructions in a few hours — Reply to this email directly, view it on GitHub <#58 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALZRDBYTPLM36P5NBE2EAXT4M4U4TAVCNFSM6AAAAACVKAGJIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMZVG4YTQNBRGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[kyuz0]

@vincentsider by using llama.cpp with the pwilkins autoparser branch merged. i can drop you the instructions in a few hours

when you have a sec, can you also point me to that branch? I guess it's not merged because of some code quality issues or just on-going testing?

[jwefers]

@vincentsider clone this repo, add these new lines to the Dockerfile.rocm7-nightlies after line 60:

RUN git clone -b ${BRANCH} --single-branch --recursive ${REPO} . \
   && git clean -xdf \
   && git submodule update --recursive
+RUN git remote add pwilkin https://github.com/pwilkin/llama.cpp.git \
+  && git config user.name "nope" && git config user.email "no@nope.com" \
+  && git fetch pwilkin autoparser \
+  && git merge pwilkin/autoparser

then build the docker image locally, e.g. sudo docker build toolboxes -f toolboxes/Dockerfile.rocm7-nightlies or however, maybe using be toolbox commands, depends on your environment.