Trying to find the kernel commit which makes WSL non-responsive

[carlfriedrich]

We're trying to find the kernel commit which makes WSL non-responsive after hibernation, which is described in the issues microsoft/WSL#8696 and microsoft/WSL#6982.

Our starting point

• @burk3 has described here how to build and use a custom kernel based on the linux-msft-5.4.72 tag. I have followed these steps and did not have the issue within over a year.
• @onereal7 tried other versions and reported here the first tag showing the issue is v5.5-rc1. I have built that version as well and can confirm this.
• The common base of these two tags is v5.4. We assume that the issue does not appear in this version. This has to be confirmed, though.

Bisecting the kernel

We have about 13,000 commits between v5.4 and v5.5-rc1. Using git bisect we should be able to track down the commit introducing the issue within 14 rounds. As a start, I have built the start and end versions and one in between. I will update this table as soon as the versions are confirmed to be working or non-working and add new versions as I continue the bisection. The links in the table lead to the release page for the corresponding version where you can download the kernel image.

Kernel version	Good	Reports good / bad
v5.4	✅	6 / 0
v5.4-2622-g386403a115f	✅	5 / 0
v5.4-2759-ga86f69d3349	✅	8 / 0
v5.4-2809-ga25bbc2644f	✅	4 / 0
v5.4-2816-gcd4771f7709	✅	5 / 0
v5.4-2819-g64d6a12094f	❌	0 / 3
v5.4-2824-g24ee25a6da8	❌	1 / 4
v5.4-2841-gda42761df5c	❌	1 / 4
v5.4-2929-g1d87200446f	❌	0 / 4
v5.4-3127-g77a05940eee	❌	0 / 2
v5.4-3434-g3f1b210a7f9	❌	0 / 2
v5.4-4535-g9a3d7fd275b	❌	0 / 2
v5.5-rc1	❌	0 / 2

How you can help

• Subscribe to this issue to stay up to date about the bisection.
• Download the current test version (the one with a ❔ in the above table) and set it up in your WSL instance like described in the README.
• Leave a comment in this issue, either
  • when WSL becomes unresponsive using this kernel, or
  • if you don't encounter the issue within a week using this kernel.
• In both cases, include the output of uname --kernel-release in your comment.

I will wait for a reasonable number of reports for each version, so even if somebody else reported a working or non-working version before, please do report your experience as well.

How you cannot help

We're not looking for any workarounds or environment information related to the issue here. I am not a Microsoft developer, so I am not debugging the issue or collecting any information to help solving it.
If you want to share any information of this kind, please do so in one of the upstream issues.

Thanks a lot for your help in advance. 💚

---

Update

We have found the kernel commit introducing the issue:

Merge commit:
microsoft/WSL2-Linux-Kernel@64d6a12094f3

Atomic commit:
microsoft/WSL2-Linux-Kernel@dce7cd62754b5

From here on I will try to build more recent kernel versions with the commit reverted. Feel free to use these and report your experience.

Kernel version	Good	Reports good / bad	Notes
v5.5-rc1-1-g0622e5f6a3	✅	3 / 0	v5.5-rc1 with 64d6a12094f3 reverted
linux-msft-wsl-5.10.102.2	✅	3 / 0	linux-msft-wsl-5.10.102.1 with dce7cd62754b5 reverted
linux-msft-wsl-5.15.153.2	✅	6 / 0	linux-msft-wsl-5.15.153.1 with dce7cd62754b5 reverted
linux-msft-wsl-6.6.75.1	❔	0 / 0	linux-msft-wsl-6.6.75.1 official Microsoft WSL2 kernel with bugfix

[unwiredben]

Installed v5.4-4535-g9a3d7fd275b on my laptop this morning and hibernated it while traveling to my office. After about an hour of use after coming out of hibernation, I hit the unresponsive/high-CPU-usage issue and needed to kill WSL service to recover.

[carlfriedrich]

@unwiredben Thanks a lot for testing it out, that is really helpful! Can you also check if v5.4 is working for you?

[unwiredben]

@unwiredben Thanks a lot for testing it out, that is really helpful! Can you also check if v5.4 is working for you?

I just switched over to 5.4 and will report back in a few days unless I see if hang first.

[onereal7]

@carlfriedrich, nice setup you have here! :)
I wish I could contribute more now, however a Win10 update a month ago broke my hibernation at all so it now almost always acts as a regular shutdown..

[carlfriedrich]

@carlfriedrich, nice setup you have here! :) I wish I could contribute more now, however a Win10 update a month ago broke my hibernation at all so it now almost always acts as a regular shutdown..

Well, then one might say the update fixed the issue for you. 😋

[unwiredben]

So far, no hangs with 5.4 across three hibernate cycles.

[mannfuri]

Which CPU do you guys use, AMD or INTEL?
The wsl kernel versions of the computers at my company and at home are the same, both are the latest official versions of WSL.
The computer at the company has not had a CPU 100% issue for a long time, but the computer at home still frequently encounters this problem.
The computer at the company uses an INTEL CPU, while the one at my home uses an AMD CPU.

[mannfuri]

switched to 5.4 today
I will come back to provide feedback in a while.

[carlfriedrich]

Which CPU do you guys use, AMD or INTEL? The wsl kernel versions of the computers at my company and at home are the same, both are the latest official versions of WSL. The computer at the company has not had a CPU 100% issue for a long time, but the computer at home still frequently encounters this problem. The computer at the company uses an INTEL CPU, while the one at my home uses an AMD CPU.

@mannfuri Thanks for your feedback. That's quite interesting, actually. I am on Intel on both my work and my home machine, and I get the issue on both. So AMD vs. Intel does not seem to be responsible for the issue to appear. I remember someone reporting in the upstream issue, that they also get the issue on ARM.
There must be some component, though, which makes a difference. According to the comments from Microsoft in the upstream issue, they weren't able to reproduce the issue in any of their environments.
So that's why we - the affected users - are trying to find the bad kernel commit here. We hope that this gives Microsoft a hint where to look at, and maybe we also find out why it happens only on some machines.
Hence I very appreciate that you join our testing. Thanks a lot!

[tobyvinnell]

I've just had the usual hang with the current 5.15 kernel version today. I'm keen to help with this effort and have switched to 5.4.0 just now. I'll give that a few days before moving on to v5.4-4535

[carlfriedrich]

@tobyvinnell Great, thanks a lot for your help!