Add static route check in warm reboot

[xincunli-sonic]

Description of PR

Summary

Adds comprehensive tests to validate static route persistence and traffic forwarding during warmboot and config reload scenarios.

Fixes: sonic-net/sonic-buildimage#21423

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• New Test case
  • Skipped for non-supported platforms

Back port request

• 202205
• 202305
• 202311
• 202405
• 202411
• 202505
• 202511

Approach

What is the motivation for this PR?

Static routes were not being validated during warmboot and config reload operations, leading to issues where routes could be lost or traffic forwarding could fail after these critical operations. Issue #21423 reported that static routes were not persisting correctly through warmboot, causing network connectivity problems.

How did you do it?

Added 4 new comprehensive test functions to tests/route/test_static_route.py:

1. test_static_route_warmboot - Tests IPv4 static route persistence through warmboot

   • Validates route configuration before warmboot
   • Performs warmboot with finalizer wait
   • Verifies routes persist and traffic forwarding works after warmboot
   • Confirms BGP route advertisements are maintained

2. test_static_route_ecmp_warmboot - Tests IPv4 ECMP static routes with multiple nexthops through warmboot

   • Validates all ECMP paths are configured correctly
   • Verifies load-balancing across nexthops before warmboot
   • Confirms all paths persist through warmboot
   • Validates traffic continues across all paths post-warmboot

3. test_static_route_ipv6_warmboot - Tests IPv6 static route persistence through warmboot

   • Full IPv6 static route lifecycle validation
   • NDP entry management during warmboot
   • IPv6 traffic forwarding verification

4. test_static_route_config_reload_with_traffic - Tests static route persistence through config reload

   • Comprehensive traffic validation before and after reload
   • Handles dual-TOR mux state changes
   • Validates BGP convergence post-reload

Each test follows a complete lifecycle:

• Setup: Configure routes and traffic responders
• Pre-operation validation: Verify routes in kernel, test traffic, check BGP advertisements
• Operation: Perform warmboot or config reload
• Post-operation validation: Verify route persistence, BGP convergence, traffic forwarding
• Cleanup: Remove routes and verify cleanup

How did you verify/test it?

In Lab:

tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
/opt/venv/lib/python3.12/site-packages/pytest_ansible/host_manager/v213.py:13: DeprecationWarning: Host management is deprecated and will be removed in a future release
class HostManagerV213(BaseHostManager):

tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
/opt/venv/lib/python3.12/site-packages/ansible/plugins/loader.py:1485: UserWarning: AnsibleCollectionFinder has already been configured
warnings.warn('AnsibleCollectionFinder has already been configured')

tests/route/test_static_route.py: 186 warnings
/usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=79633) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()

tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
/var/src/sonic-mgmt-int/tests/conftest.py:1363: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
record_testsuite_property("timestamp", datetime.utcnow())

tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
/var/src/sonic-mgmt-int/tests/common/dualtor/mux_simulator_control.py:705: DeprecationWarning: Deprecated toggle fixture, please use setup_dualtor_mux_ports (docs/tests/setup.dualtor.mux.ports.md).
warnings.warn("Deprecated toggle fixture, please use setup_dualtor_mux_ports "

tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
tests/route/test_static_route.py::test_static_route_warmboot[str2-7260cx3-acs-11]
/opt/venv/lib/python3.12/site-packages/ptf/mask.py:69: DeprecationWarning: "set_do_not_care_scapy" is going to be deprecated, please switch to the new one: "set_do_not_care_packet"
warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------------------------------------------------------------------- generated xml file: /var/src/sonic-mgmt-int/tests/logs/route/test_static_route.py::test_static_route_warmboot.xml -----------------------------------------------------------------------

Results (716.58s (0:11:56)):
1 passed
DEBUG:tests.conftest:[log_custom_msg] item: <Function test_static_route_warmboot[str2-7260cx3-acs-11]>
INFO:root:Can not get Allure report URL. Please check logs

yanmo@sonic-mgmt-yanmo:/var/src/sonic-mgmt-int/tests$ cat /var/src/sonic-mgmt-int/tests/logs/route/test_static_route.py::test_static_route_warmboot.xml

<?xml version="1.0" encoding="utf-8"?>
<testsuites name="pytest tests">
    <testsuite name="pytest" errors="0" failures="0" skipped="0" tests="1" time="716.565"
        timestamp="2026-01-13T20:00:55.209590+00:00" hostname="sonic-mgmt-yanmo">
        <properties>
            <property name="topology" value="dualtor-120" />
            <property name="testbed" value="vms26-dual-t0-7260" />
            <property name="timestamp" value="2026-01-13 20:05:00.589302" />
            <property name="host" value="str2-7260cx3-acs-10" />
            <property name="asic" value="broadcom" />
            <property name="platform" value="x86_64-arista_7260cx3_64" />
            <property name="hwsku" value="Arista-7260CX3-D108C8" />
            <property name="os_version" value="20251110.01" />
        </properties>
        <testcase classname="tests.route.test_static_route"
            name="test_static_route_warmboot[str2-7260cx3-acs-11]" time="530.665">
            <properties>
                <property name="start" value="2026-01-13 20:04:01.034842" />
                <property name="end" value="2026-01-13 20:12:51.705748" />
            </properties>
        </testcase>
    </testsuite>
</testsuites>

Any platform specific information?

• Tests are marked with pytest.mark.device_type('vs') for virtual switch support
• Platform-specific wait times are handled (e.g., x86_64-cel_e1031-r0 requires longer wait)
• Dual-TOR specific logic included for mux state management
• Tests use existing infrastructure and should work across all supported platforms

Supported testbed topology if it's a new test case?

• t0 - Primary topology
• m0 - Supported
• mx - Supported
• dualtor - Fully supported with specific handling for mux cable states

Tests are compatible with both single-TOR and dual-TOR configurations.

Documentation

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[bingwang-ms]

Hi @xincunli-sonic , can you add a snapshot of test result from at least 1 physical testbed since this is a new test module?