[Resolved] NVIDIA driver performance issues

[w-e-w]

Update (2023-10-31)

This issue should now be entirely resolved. NVIDIA has made a help article to disable the system memory fallback behavior. Please upgrade to the latest driver (546.01 or newer) and follow the guide on their website: https://nvidia.custhelp.com/app/answers/detail/a_id/5490

Update (2023-10-19)

issue has been reopened as it seems like more and more reports are saying that the issue is not yet fixed

Update (2023-10-17)

there seems to be some reports saying that the issue is still not fixed

comments

#11063 (comment)
#11063 (comment)

Update (2023-10-14)

This issue has reportedly been fixed by NVIDIA as of 537.58 (537.42 if using Studio release). Please update your drivers to this version or later.

The original issue description follows.

---

Discussed in #11062

^{Originally posted by w-e-w June 7, 2023}
some users have reported some issues related to the latest Nvidia drivers
nVidia drivers change in memory management vladmandic#1285
#11050 (comment)
if you have been experiencing generation slowdowns or getting stuck, consider downgrading to driver version 531 or below
NVIDIA Driver Downloads

This issue will be closed when NVIDIA resolves this issue. It currently has a tracking number of [4172676]

[tusharbhutt]

Funny, I am randomly getting the issues where an output is stuck at 50% for an hour, and I am on 531.41 for an NVIDIIA 3060 12GB model

[chaewai]

Strangely mine seems to go at normal speed for the first gen on a checkpoint, or if I change the clip on a checkpoint, but subsequent gens go muuuch slower. Annoyingly Diablo won't run on 531.

[designborg]

I can confirm this bug. I was getting results (as expected) before I installed the latest Titan RTX drivers. I will try installing a previous build.

[AIDungeonTester2]

Strangely mine seems to go at normal speed for the first gen on a checkpoint, or if I change the clip on a checkpoint, but subsequent gens go muuuch slower. Annoyingly Diablo won't run on 531.

Yeah, that's exactly how it is for me. When I tried inpainting, the first gen runs through just fine, but any subsequent ones have massive hang-ups, necessitating a restart of the commandline window and rerunning webui-user.bat.

[younyokel]

I wasn't sure if there was a problem with the drivers so I reinstalled WebUI, but the problem didn't go away. To think, everything generates fine like before, but once the High Res Fix starts and finishes, it looks like a minute pause.
Edit: confirming. Downgraded to 531.68. Now everything as it was

[hearmeneigh]

If you are stuck with a newer Nvidia driver version, downgrading to Torch 1.13.1 seems to work too.

1. Add the following to webui-user.bat:
   set TORCH_COMMAND=pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
2. Remove <webui-root>/venv directory
3. (Re)start WebUI

[Shawdooow]

I am having the opposite issue where on the newer drivers my first image generation is slow because of some clogged memory on my GPU which frees itself as soon as it gets to the second one.

Downgrading Torch didn't seem to help at all.
Downgrading from 536.23 to 531.79 fixes the problem instantly.

[younyokel]

Anyone, is this problem still relevant?

[designborg]

I haven't tried with the latest drivers, so I don't know if this issue is still ongoing.