Upgrade smartswitch via gNOI testcases by ryanzhu706 · Pull Request #22393 · sonic-net/sonic-mgmt

ryanzhu706 · 2026-02-12T20:59:45Z

Description of PR

The new test verifies that a gNOI-triggered upgrade results in an observable system reboot by checking expected reboot indicators (e.g. service downtime and CLI session interruption). This helps catch cases where an upgrade completes without actually triggering a full reboot.
This test improves coverage for SmartSwitch upgrade scenarios and prevents false positives where upgrades appear successful but reboot is skipped.

Summary:
Fixes # (issue)

Type of change

Back port request

Approach

What is the motivation for this PR?

Address test gap of smartswitch cold upgrade via gNOI.

How did you do it?

Added new testcases for smartswitch upgrade via gNOI.

How did you verify/test it?

ryanzhu@sonic-mgmt-int:/data/sonic-mgmt-int/tests$ sudo ./run_tests.sh -i ../ansible/str3,../ansible/veos -n vms66-t1-8102-7 -u -m individual -c upgrade_path/test_upgrade_smart_switch_gnoi.py -u -e "--upgrade_type=cold --base_image_list=http://10.201.148.43/pipelines/Networking-acs-buildimage-Official/pensando/internal-202506/sonic-pensando.bin --target_image_list=http://10.201.148.43/pipelines/Networking-acs-buildimage-Official/pensando/internal-202506/sonic-pensando.bin --target_version=SONiC-OS-internal-202506.154741893-16d12968f4" -d str3-8102-07 -e "--skip_sanity"
pkill: pattern that searches for process name longer than 15 characters will result in zero matches
Try `pkill -f' option to match against the complete command line.
=== Clearing pytest cache ===
=== Running tests individually ===
Running: python3 -m pytest upgrade_path/test_upgrade_smart_switch_gnoi.py --inventory ../ansible/str3,../ansible/veos --host-pattern str3-8102-07 --dpu-pattern None --testbed vms66-t1-8102-7 --testbed_file /data/sonic-mgmt-int/ansible/testbed.yaml --log-cli-level warning --log-file-level debug --kube_master unset --showlocals --assert plain --show-capture no -rav --allow_recover --ignore=ptftests --ignore=acstests --ignore=saitests --ignore=scripts --ignore=k8s --ignore=sai_qualify --log-file logs/upgrade_path/test_upgrade_smart_switch_gnoi.log --junitxml=logs/upgrade_path/test_upgrade_smart_switch_gnoi.xml --upgrade_type=cold --base_image_list=http://10.201.148.43/pipelines/Networking-acs-buildimage-Official/pensando/internal-202506/sonic-pensando.bin --target_image_list=http://10.201.148.43/pipelines/Networking-acs-buildimage-Official/pensando/internal-202506/sonic-pensando.bin --target_version=SONiC-OS-internal-202506.154741893-16d12968f4 --skip_sanity
Test session starts (platform: linux, Python 3.12.3, pytest 9.0.2, pytest-sugar 1.1.1)
ansible: 2.18.12
rootdir: /data/sonic-mgmt-int
configfile: pyproject.toml
plugins: stress-1.0.1, ansible-25.12.0, sugar-1.1.1, metadata-3.1.1, plus-0.8.1, xdist-3.8.0, cov-7.0.0, allure-pytest-2.15.3, html-4.2.0, repeat-0.9.4
collecting ... [WARNING]: While constructing a mapping from /data/sonic-mgmt-int/ansible/str3,
line 146, column 9, found a duplicate dict key (pdu-85). Using last defined
value only.
[WARNING]: Found variable using reserved name: serial
[WARNING]: While constructing a mapping from /data/sonic-mgmt-int/ansible/str3,
line 146, column 9, found a duplicate dict key (pdu-85). Using last defined
value only.
[WARNING]: Found variable using reserved name: serial
[WARNING]: While constructing a mapping from /data/sonic-mgmt-int/ansible/str3,
line 146, column 9, found a duplicate dict key (pdu-85). Using last defined
value only.
[WARNING]: Found variable using reserved name: serial
[WARNING]: While constructing a mapping from /data/sonic-mgmt-int/ansible/str3,
line 146, column 9, found a duplicate dict key (pdu-85). Using last defined
value only.
[WARNING]: Found variable using reserved name: serial
[WARNING]: While constructing a mapping from /data/sonic-mgmt-int/ansible/str3,
line 146, column 9, found a duplicate dict key (pdu-85). Using last defined
value only.
[WARNING]: Found variable using reserved name: serial
collected 2 items


--------------------------------------------------------- live log teardown ---------------------------------------------------------
WARNING  tests.common.plugins.memory_utilization.memory_utilization:memory_utilization.py:64 Skipping memory check for docker-radv due to zero value

WARNING  tests.common.plugins.memory_utilization.memory_utilization:memory_utilization.py:64 Skipping memory check for docker-teamd due to zero value
 tests/upgrade_path/test_upgrade_smart_switch_gnoi.py::test_upgrade_one_dpu_via_gnoi[str3-8102-07] ✓                   50% █████
 tests/upgrade_path/test_upgrade_smart_switch_gnoi.py::test_upgrade_multiple_dpus_via_gnoi_parallel[str3-8102-07] s   100% ██████████

Any platform specific information?

No

Supported testbed topology if it's a new test case?

smartswitch

Documentation

mssonicbld · 2026-02-12T20:59:53Z

/azp run

azure-pipelines · 2026-02-12T21:00:12Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-13T00:05:17Z

/azp run

azure-pipelines · 2026-02-13T00:05:32Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-20T18:55:19Z

/azp run

azure-pipelines · 2026-02-20T18:55:26Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

mssonicbld · 2026-02-27T23:23:41Z

/azp run

azure-pipelines · 2026-02-27T23:23:47Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

hdwhdw

AI Agent on behalf of Dawei:

Thanks for adding the SmartSwitch gNOI upgrade test coverage — this fills an important gap. A few thoughts below, mostly around one design suggestion and a couple of smaller items.

tests/common/helpers/upgrade_helpers.py

tests/upgrade_path/test_upgrade_smart_switch_gnoi.py

tests/common/ptf_grpc.py

mssonicbld · 2026-03-02T22:40:52Z

/azp run

azure-pipelines · 2026-03-02T22:41:08Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-02T23:01:26Z

/azp run

azure-pipelines · 2026-03-02T23:01:42Z

Azure Pipelines successfully started running 1 pipeline(s).

tests/common/fixtures/grpc_fixtures.py

mssonicbld · 2026-03-10T18:33:03Z

/azp run

azure-pipelines · 2026-03-10T18:33:19Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-10T20:02:57Z

/azp run

azure-pipelines · 2026-03-10T20:03:13Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-11T00:21:27Z

/azp run

azure-pipelines · 2026-03-11T00:21:42Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-11T17:22:17Z

/azp run

azure-pipelines · 2026-03-11T17:22:34Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-16T17:04:57Z

/azp run

azure-pipelines · 2026-03-16T17:05:13Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-16T17:10:48Z

/azp run

azure-pipelines · 2026-03-16T17:11:06Z

Azure Pipelines successfully started running 1 pipeline(s).

vaibhavhd

Added non blocking comments.

vaibhavhd · 2026-03-16T21:21:25Z

tests/upgrade_path/test_upgrade_smart_switch_gnoi.py

+
+    ss_target_index = request.config.getoption("ss_target_index")          # int
+    ss_target_indices = request.config.getoption("ss_target_indices")      # "0,1,2,3"
+    ss_reboot_ready_timeout = 600


Non blocking - please justify why 600 seconds is set to reboot ready timeout? What factors govern or influence this decision?

vaibhavhd · 2026-03-16T21:22:24Z

tests/upgrade_path/test_upgrade_smart_switch_gnoi.py

+    if ss_target_index in (None, ""):
+        ss_target_index = 3
+    if not ss_reboot_ready_timeout:
+        ss_reboot_ready_timeout = 1200


Non blocking - Why bumping to 1200s?

vaibhavhd · 2026-03-16T21:24:42Z

tests/upgrade_path/test_upgrade_smart_switch_gnoi.py

+    # Best-effort cleanup (may run on NPU; harmless)
+    duthost.shell(f"rm -f {dut_image_path}", module_ignore_errors=True)


Considering that test has to represent what happens in production (real worls), keep in mind if you do need this step (even if this is optional) you need to

make an API out of it (as the client may not have ability to run CLIs)

formulize it as a step of upgrade MoP.

vaibhavhd · 2026-03-16T21:28:21Z

tests/common/helpers/upgrade_helpers.py

+    return {"transfer_resp": transfer_resp, "setpkg_resp": setpkg_resp, "reboot_resp": reboot_resp}
+
+
+def perform_gnoi_upgrade_smartswitch_dpu(duthost, tbinfo, ptf_gnoi, cfg: GnoiUpgradeConfig) -> Dict:


Optional - Considering the concurrent call has a name perform_gnoi_upgrade_smartswitch_dpus_parallel, a better name for this function should be perform_gnoi_upgrade_smartswitch_single_dpu?

vaibhavhd · 2026-03-16T21:29:32Z

tests/common/helpers/upgrade_helpers.py

+    if cfg.allow_fail:
+        return {"transfer_resp": transfer_resp, "setpkg_resp": setpkg_resp, "reboot_resp": reboot_resp}
+
+    ok = _wait_gnoi_time_ready(ptf_gnoi, md, cfg)


@v-cshekar , as a next step from here, please include your changes (generic perform_reboot call in this section)

vaibhavhd · 2026-03-16T21:30:58Z

tests/common/ptf_gnoi.py

            "SetPackage via gNOI System.SetPackage (streaming): filename=%s version=%s activate=%s",
            local_path, version, activate,
        )
+        self.grpc_client.configure_max_time(3600)        # allow long SetPackage


Why is this needed?

grpc client and server connections dropped when "set_package" which is image installation. It complains about too many pings and "ENHANCE_YOUR_CALM" and disconnected, considering it's client/server side session time out issue, added the max_time and keep_alive parameters in the grpc command.

vaibhavhd · 2026-03-16T21:31:17Z

tests/common/ptf_gnoi.py

            local_path, version, activate,
        )
+        self.grpc_client.configure_max_time(3600)        # allow long SetPackage
+        self.grpc_client.configure_keepalive_time(300)   # 5 min keepalive (less aggressive


Same here - why is this step really needed?

We really need to understand this step clearly - if this is client side need, or server's limitation -- @hdwhdw

If this is client side need, we need to update the MoP that we have for our clients.

We maybe need some changes on server side to determine when the installation is completed.

vaibhavhd · 2026-03-16T21:35:04Z

tests/upgrade_path/test_upgrade_smart_switch_gnoi.py

+
+
+@pytest.mark.device_type("smartswitch")
+def test_upgrade_multiple_dpus_via_gnoi_parallel(


This testcase is not yet tested. The results are trusted only from single DPU upgrade.

Still allowing this to go in in favor of having test coverage out there.

The new test verifies that a gNOI-triggered upgrade results in an observable system reboot by checking expected reboot indicators (e.g. service downtime and CLI session interruption). This helps catch cases where an upgrade completes without actually triggering a full reboot. This test improves coverage for SmartSwitch upgrade scenarios and prevents false positives where upgrades appear successful but reboot is skipped. Tests - Passed - tests/upgrade_path/test_upgrade_smart_switch_gnoi.py::test_upgrade_one_dpu_via_gnoi[str3-8102-07] ✓ 50% Skipped - tests/upgrade_path/test_upgrade_smart_switch_gnoi.py::test_upgrade_multiple_dpus_via_gnoi_parallel[str3-8102-07] Signed-off-by: Abhishek <[email protected]>

The new test verifies that a gNOI-triggered upgrade results in an observable system reboot by checking expected reboot indicators (e.g. service downtime and CLI session interruption). This helps catch cases where an upgrade completes without actually triggering a full reboot. This test improves coverage for SmartSwitch upgrade scenarios and prevents false positives where upgrades appear successful but reboot is skipped. Tests - Passed - tests/upgrade_path/test_upgrade_smart_switch_gnoi.py::test_upgrade_one_dpu_via_gnoi[str3-8102-07] ✓ 50% Skipped - tests/upgrade_path/test_upgrade_smart_switch_gnoi.py::test_upgrade_multiple_dpus_via_gnoi_parallel[str3-8102-07] Signed-off-by: Venkata Gouri Rajesh Etla <[email protected]>

Added DPU upgrade testcases.

9380a9a

github-actions bot requested review from lolyu, wangxin and yutongzhang-microsoft February 12, 2026 21:00

Added dpu upgrade via gnoi testcases.

002f82f

address comments

428d3f7

vvolam requested a review from gpunathilell February 26, 2026 21:36

Modify tests

8901ed8

ryanzhu706 changed the title ~~Added DPU upgrade testcases.~~ Upgrade smartswitch via gNOI testcases Feb 27, 2026

hdwhdw self-requested a review March 1, 2026 06:01

hdwhdw reviewed Mar 1, 2026

View reviewed changes

tests/common/helpers/upgrade_helpers.py Outdated Show resolved Hide resolved

tests/upgrade_path/test_upgrade_smart_switch_gnoi.py Show resolved Hide resolved

tests/common/ptf_grpc.py Show resolved Hide resolved

ryanzhu706 added 2 commits March 2, 2026 22:37

changes

0df33ae

Fix conflicts

146b31b

pre-commit check

7dfd6ad

hdwhdw reviewed Mar 6, 2026

View reviewed changes

tests/common/fixtures/grpc_fixtures.py Outdated Show resolved Hide resolved

Address comments

adf6fd2

ryanzhu706 requested a review from hdwhdw March 10, 2026 18:33

Address comments

42b8242

address comments

50f7f13

ryanzhu706 requested a review from vaibhavhd March 11, 2026 00:27

Reference smartswitch reboot functions.

f330cb2

vvolam requested review from nhe-NV and rameshraghupathy March 12, 2026 20:35

Address comments

a032e3d

github-actions bot requested review from rawal01 and xwjiang-ms March 16, 2026 17:05

Address comments

1e85d11

hdwhdw approved these changes Mar 16, 2026

View reviewed changes

vaibhavhd approved these changes Mar 16, 2026

View reviewed changes

vaibhavhd merged commit c9fe85a into sonic-net:master Mar 16, 2026
15 of 16 checks passed

vaibhavhd added the Request for 202511 branch Request to backport a change to 202511 branch label Mar 16, 2026

ryanzhu706 deleted the dpu_upgrade branch March 24, 2026 22:49

		# Best-effort cleanup (may run on NPU; harmless)
		duthost.shell(f"rm -f {dut_image_path}", module_ignore_errors=True)

		return {"transfer_resp": transfer_resp, "setpkg_resp": setpkg_resp, "reboot_resp": reboot_resp}


		def perform_gnoi_upgrade_smartswitch_dpu(duthost, tbinfo, ptf_gnoi, cfg: GnoiUpgradeConfig) -> Dict:



		@pytest.mark.device_type("smartswitch")
		def test_upgrade_multiple_dpus_via_gnoi_parallel(

Conversation

ryanzhu706 commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Feb 12, 2026

Uh oh!

azure-pipelines bot commented Feb 12, 2026

Uh oh!

mssonicbld commented Feb 13, 2026

Uh oh!

azure-pipelines bot commented Feb 13, 2026

Uh oh!

mssonicbld commented Feb 20, 2026

Uh oh!

azure-pipelines bot commented Feb 20, 2026

Uh oh!

mssonicbld commented Feb 27, 2026

Uh oh!

azure-pipelines bot commented Feb 27, 2026

Uh oh!

hdwhdw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Mar 2, 2026

Uh oh!

azure-pipelines bot commented Mar 2, 2026

Uh oh!

mssonicbld commented Mar 2, 2026

Uh oh!

azure-pipelines bot commented Mar 2, 2026

Uh oh!

Uh oh!

mssonicbld commented Mar 10, 2026

Uh oh!

azure-pipelines bot commented Mar 10, 2026

Uh oh!

mssonicbld commented Mar 10, 2026

Uh oh!

azure-pipelines bot commented Mar 10, 2026

Uh oh!

mssonicbld commented Mar 11, 2026

Uh oh!

azure-pipelines bot commented Mar 11, 2026

Uh oh!

mssonicbld commented Mar 11, 2026

Uh oh!

azure-pipelines bot commented Mar 11, 2026

Uh oh!

mssonicbld commented Mar 16, 2026

Uh oh!

azure-pipelines bot commented Mar 16, 2026

Uh oh!

mssonicbld commented Mar 16, 2026

Uh oh!

azure-pipelines bot commented Mar 16, 2026

Uh oh!

vaibhavhd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanzhu706 commented Feb 12, 2026 •

edited

Loading