feat: new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade by cyw233 · Pull Request #19263 · sonic-net/sonic-mgmt

cyw233 · 2025-06-30T07:40:19Z

Description of PR

We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread.

To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations.

Summary:
Fixes # (issue) Microsoft ADO 33039693

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld · 2025-06-30T07:40:27Z

/azp run

azure-pipelines · 2025-06-30T07:40:42Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-07-11T11:08:58Z

/azp run

azure-pipelines · 2025-07-11T11:09:12Z

Azure Pipelines successfully started running 1 pipeline(s).

cyw233 · 2025-07-12T03:57:50Z

azure-pipelines.yml

          MAX_WORKER: $(INSTANCE_NUMBER)
          KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
          MGMT_BRANCH: $(BUILD_BRANCH)
+          COMMON_EXTRA_PARAMS: "--disable_sai_validation "


Temporarily disable SAI validation for now as it will not be compatible with Ubuntu 24.04 due to the usage of concurrent.futures. We will refactor the SAI validation and re-enable it later. Microsoft ADO to track the progress: 33758029

…untu 24.04 upgrade (#19599) What is the motivation for this PR? #19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade

liuh-80 · 2025-07-15T13:19:32Z

@cyw233 . all sonic-buildimage master branch PR validation failed with following message, I guess it's related with this PR:

File "/var/src/sonic-mgmt/tests/common/helpers/multi_thread_utils.py", line 47, in _wrapper
raise RuntimeError("Thread worker aborted: " + repr(be))
RuntimeError: Thread worker aborted: AttributeError("'ApplyResult' object has no attribute '_condition'")

cyw233 · 2025-07-15T22:42:49Z

@cyw233 . all sonic-buildimage master branch PR validation failed with following message, I guess it's related with this PR:

File "/var/src/sonic-mgmt/tests/common/helpers/multi_thread_utils.py", line 47, in _wrapper raise RuntimeError("Thread worker aborted: " + repr(be)) RuntimeError: Thread worker aborted: AttributeError("'ApplyResult' object has no attribute '_condition'")

Hey @liuh-80, yeah we need to temporarily disable SAI validation for now as it will not be compatible with the change due to the usage of concurrent.futures. We will refactor the SAI validation and re-enable it later. Microsoft ADO to track the progress: 33758029

Therefore, I added the --disable_sai_validation option to the azure pipeline definition: sonic-net/sonic-buildimage#23346

Why I did it Temporarily disable SAI validation for now as it is not compatible with the change in sonic-net/sonic-mgmt#19263 due to the usage of concurrent.futures. We will refactor the SAI validation and re-enable it later. Microsoft ADO to track the progress: 33758029. Work item tracking Microsoft ADO (number only): 33039693 signed-off-by: jianquanye@microsoft.com

…19263) Description of PR We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread. To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations. Summary: Fixes # (issue) Microsoft ADO 33039693 signed-off-by: jianquanye@microsoft.com

…untu 24.04 upgrade (sonic-net#19599) What is the motivation for this PR? sonic-net#19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade

…19263) Description of PR We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread. To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations. Summary: Fixes # (issue) Microsoft ADO 33039693 signed-off-by: jianquanye@microsoft.com

…untu 24.04 upgrade (sonic-net#19599) What is the motivation for this PR? sonic-net#19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade

…19263) Description of PR We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread. To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations. Summary: Fixes # (issue) Microsoft ADO 33039693 signed-off-by: jianquanye@microsoft.com

…untu 24.04 upgrade (sonic-net#19599) What is the motivation for this PR? sonic-net#19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade

According to #19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR #21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in #19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR #21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com>

…19263) Description of PR We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread. To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations. Summary: Fixes # (issue) Microsoft ADO 33039693 signed-off-by: jianquanye@microsoft.com Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>

…untu 24.04 upgrade (sonic-net#19599) What is the motivation for this PR? sonic-net#19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com> Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>

…19263) Description of PR We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread. To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations. Summary: Fixes # (issue) Microsoft ADO 33039693 signed-off-by: jianquanye@microsoft.com Signed-off-by: Aharon Malkin <amalkin@nvidia.com>

…untu 24.04 upgrade (sonic-net#19599) What is the motivation for this PR? sonic-net#19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade Signed-off-by: Aharon Malkin <amalkin@nvidia.com>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com> Signed-off-by: Aharon Malkin <amalkin@nvidia.com>

…19263) Description of PR We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread. To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations. Summary: Fixes # (issue) Microsoft ADO 33039693 signed-off-by: jianquanye@microsoft.com Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>

…untu 24.04 upgrade (sonic-net#19599) What is the motivation for this PR? sonic-net#19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com> Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>

…19263) Description of PR We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread. To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations. Summary: Fixes # (issue) Microsoft ADO 33039693 signed-off-by: jianquanye@microsoft.com

…untu 24.04 upgrade (sonic-net#19599) What is the motivation for this PR? sonic-net#19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com> Signed-off-by: YiFan Wang <yifan@nexthop.ai>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com>

According to #19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR #21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in #19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR #21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com> Signed-off-by: Priyansh Tratiya <ptratiya@microsoft.com>

…19263) Description of PR We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread. To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations. Summary: Fixes # (issue) Microsoft ADO 33039693 signed-off-by: jianquanye@microsoft.com Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>

…untu 24.04 upgrade (sonic-net#19599) What is the motivation for this PR? sonic-net#19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade Signed-off-by: Guy Shemesh <gshemesh@nvidia.com>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com> Signed-off-by: Lakshmi Yarramaneni <lakshmi@nexthop.ai>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com>

…19263) Description of PR We will soon upgrade sonic-mgmt docker to Ubuntu 24.04 which comes with Python 3.12 + Ansible 2.18.6. Since Python 3.12 enforces more rigorous checks around fork() in multi‐threaded programs, we will start getting the ansible.errors.AnsibleError: A worker was found in a dead state exception due to ThreadPoolExecutor from concurrent.futures.thread. To mitigate this issue, we re-implemented the SafeThreadPoolExecutor class with the traditional ThreadPool from multiprocessing.pool for multithreading operations. Summary: Fixes # (issue) Microsoft ADO 33039693 signed-off-by: jianquanye@microsoft.com Signed-off-by: Yael Tzur <ytzur@nvidia.com>

…untu 24.04 upgrade (sonic-net#19599) What is the motivation for this PR? sonic-net#19263 added new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade and common param in PR test, but we also need the param in baseline test How did you do it? Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade Signed-off-by: Yael Tzur <ytzur@nvidia.com>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com> Signed-off-by: Yael Tzur <ytzur@nvidia.com>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com> Signed-off-by: Lakshmi Yarramaneni <lakshmi@nexthop.ai>

According to sonic-net#19263, python 3.12 enforces more rigorous check around fork() in multiple-threaded programs. After the docker-sonic-mgmt image is upgraded to Ubuntu 24.04. python and ansible are upgraded too. With python 3.12 and ansible 2.18 in new docker-sonic-mgmt, the nbrhosts fixture depends on concurrent.futures may fail with error like below: ``` self = <ansible.plugins.strategy.linear.StrategyModule object at 0x7596c07986e0> iterator = <ansible.executor.play_iterator.PlayIterator object at 0x7596c09b2a80> def _wait_on_pending_results(self, iterator): ''' Wait for the shared counter to drop to zero, using a short sleep between checks to ensure we don't spin lock ''' ret_results = [] display.debug("waiting for pending results...") while self._pending_results > 0 and not self._tqm._terminated: if self._tqm.has_dead_workers(): > raise AnsibleError("A worker was found in a dead state") E ansible.errors.AnsibleError: A worker was found in a dead state ``` PR sonic-net#21407 introduced threading lock to temporarily workaround the issue. A better way to fix the issue is to use the SafeThreadPoolExecutor updated in sonic-net#19263 to initialize the `nbrhosts` objects. This change reverted the threading lock of PR sonic-net#21407 and updated the `nbrhosts` fixture to use the new SafeThreadPoolExecutor. Signed-off-by: Xin Wang <xiwang5@microsoft.com> Signed-off-by: Raghavendran Ramanathan <rraghav@cisco.com>

cyw233 changed the title ~~Switch to new SafeThreadPoolExecutor to prepare for Ubuntu 24.04 upgrade~~ feat: use new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade Jun 30, 2025

cyw233 marked this pull request as ready for review July 11, 2025 01:31

yutongzhang-microsoft approved these changes Jul 11, 2025

View reviewed changes

feat: new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade

32020b1

cyw233 force-pushed the change-to-new-multi-thread-utils branch from 876fd04 to 32020b1 Compare July 11, 2025 11:08

cyw233 changed the title ~~feat: use new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade~~ feat: new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade Jul 11, 2025

cyw233 commented Jul 12, 2025

View reviewed changes

yejianquan merged commit 86c73fa into sonic-net:master Jul 14, 2025
18 checks passed

xwjiang-ms mentioned this pull request Jul 15, 2025

Add common param in baseline to support SafeThreadPoolExecutor for Ubuntu 24.04 upgrade #19599

Merged

11 tasks

liuh-80 mentioned this pull request Jul 15, 2025

[Orchagent] Enable ZMQ on orchagent route table. sonic-net/sonic-buildimage#23028

Merged

9 tasks

ymd-arista mentioned this pull request Jul 15, 2025

[Arista SKUs] Use more generic JSON reboot cause extra info file sonic-net/sonic-buildimage#22638

Merged

12 tasks

cyw233 mentioned this pull request Jul 15, 2025

fix: temporarily disable sai validation sonic-net/sonic-buildimage#23346

Merged

9 tasks

opcoder0 mentioned this pull request Jul 16, 2025

Bug: Sometimes running ptfhost.shell() or duthost.shell() will be stuck when running test_bfd_basic #19519

Open

ymd-arista mentioned this pull request Jul 18, 2025

Regression: acl/test_acl.py errors out with RuntimeError: Thread worker aborted exception #19712

Closed

StormLiangMS added Request for 202505 branch Approved for 202505 branch labels Nov 25, 2025

mssonicbld mentioned this pull request Jan 20, 2026

[action] [PR:21423] Fix dead worker issue by using SafeThreadPoolExecutor #22029

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade#19263

feat: new SafeThreadPoolExecutor for Ubuntu 24.04 upgrade#19263
yejianquan merged 1 commit intosonic-net:masterfrom
cyw233:change-to-new-multi-thread-utils

cyw233 commented Jun 30, 2025 •

edited

Loading

Uh oh!

mssonicbld commented Jun 30, 2025

Uh oh!

azure-pipelines bot commented Jun 30, 2025

Uh oh!

mssonicbld commented Jul 11, 2025

Uh oh!

azure-pipelines bot commented Jul 11, 2025

Uh oh!

cyw233 Jul 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

liuh-80 commented Jul 15, 2025

Uh oh!

cyw233 commented Jul 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

cyw233 commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Jun 30, 2025

Uh oh!

azure-pipelines bot commented Jun 30, 2025

Uh oh!

mssonicbld commented Jul 11, 2025

Uh oh!

azure-pipelines bot commented Jul 11, 2025

Uh oh!

cyw233 Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

liuh-80 commented Jul 15, 2025

Uh oh!

cyw233 commented Jul 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cyw233 commented Jun 30, 2025 •

edited

Loading

cyw233 Jul 12, 2025 •

edited

Loading