[fast-reboot]: New version of fast reboot test by pavel-shirshov · Pull Request #204 · sonic-net/sonic-mgmt

pavel-shirshov · 2017-06-30T02:37:16Z

Changes:

"Pings" are changed. They are sent in two direction (as previous):
- From one of vlan interfaces to T1 devices. The source ip, source interface, and destination IP are chosen randomly from valid choices. Number of packets are 100
- From all of portchannel ports to all of vlan ports. The source ip, source interface, and destination IP are chosen sequentially from valid choices.
Currently we have 500 distinct destination vlan addresses. Our target to have 1000 of them.
The test measures the dataplane disruption time (as previous), but the disruption time is configurable now
The test measures the control plane disruption time.
The test measures number of packets, which goes in both directions when there're no control plane (sometimes control plane route our packets instead of dataplane)
When the test starts fast-reboot procedure it connects to all VMs (which emulates T1) and starts fetching statuses of BGP and LACP protocols. If BGP routes goes away from VMs, or LACP interfaces goes down the test fails.
The test checks that DUT has BGP graceful restart enabled and graceful restart timeout is equal to 120 seconds.
The test checks that DUT made fast-reboot until the BGP graceful restart timeout doesn't expired
The test checks that VMs has a right next-hop interface (PortChannel), not Management1

lguohan · 2017-07-05T21:27:38Z

ansible/roles/test/files/ptftests/fast-reboot.py

+#    if LACP is inactive on VMs and interfaces goes down test failes
+#    if BGP graceful restart timeout is not equal 120 seconds the test fails
+#    if BGP graceful restart is not enabled on DUT the test fails
+#    If BGP graceful restart timeout is less than 15 seconds the test fails


If BGP graceful restart timeout is less than 15 seconds the test fails [](start = 5, length = 70)

this is contradictory with the previous statement.

"if BGP graceful restart timeout is not equal 120 seconds the test fails"

previous statement was about static,configured value of graceful restart, this one about dynamic value

lguohan · 2017-07-05T21:27:58Z

ansible/roles/test/files/ptftests/fast-reboot.py

+# 1. Check that DUT is stable. That means that "pings" work in both directions: from T1 to servers and from servers to T1.
 # 2. If DUT is stable the test starts continiously pinging DUT in both directions.
 # 3. The test runs '/usr/bin/fast-reboot' on DUT remotely. The ssh key supposed to be uploaded by ansible before the test
 # 3. As soon as it sees that ping starts failuring in one of directions the test registers a start of dataplace disruption


3 [](start = 2, length = 1)

step 4.

lguohan · 2017-07-05T21:29:54Z

ansible/roles/test/files/ptftests/fast-reboot.py

+    def parse_lacp(self, output):
+        return output.find('Bundled') != -1
+
+    def parse_bgp_neig_once(self, output):


neig? can we use the full name?

lguohan · 2017-07-05T21:40:20Z

ansible/roles/test/files/ptftests/fast-reboot.py


        return stdout, stderr, return_code

+    def ssh_job(self, ip, queue):


ssh_job [](start = 8, length = 7)

can we make the name more meanful? like peer_state_check

lguohan · 2017-07-05T21:57:46Z

ansible/roles/test/files/ptftests/fast-reboot.py


    def ping_iteration(self):
-        return self.pingFromServers() > 0 and self.pingFromUpperTier() > 0
+        nr_from_servers = self.pingFromServers()


what is nr?

it is for number

lguohan · 2017-07-05T22:00:25Z

ansible/roles/test/files/ptftests/fast-reboot.py

+                q.put('go')
            if success and is_stop or not success and not is_stop:
-                self.log("Success", True)
+                self.log("Base state", True)


Base state [](start = 26, length = 10)

what is base state and what is changed state?

We're looking for a change in this function. So "Base state" it's a state which is expected before the change, "Changed" state it's a state which is expected after change.

lguohan · 2017-07-05T22:01:48Z

ansible/roles/test/files/ptftests/fast-reboot.py

        self.log("Wait until ASIC stops")
        self.timeout(self.task_timeout, "DUT hasn't stopped in %d seconds" % self.task_timeout)
-        no_routing_start = self.check_stop()
+        no_routing_start, upper_replies = self.check_stop()


check_stop [](start = 47, length = 10)

rename to check_forwarding_stop?

lguohan · 2017-07-05T22:02:44Z

ansible/roles/test/files/ptftests/fast-reboot.py

        self.log("ASIC was stopped, Waiting until it's up. Stop time: %s" % str(no_routing_start))
        self.timeout(self.task_timeout, "DUT hasn't started to work for %d seconds" % self.task_timeout)
-        no_routing_stop = self.check_start()
+        no_routing_stop, _ = self.check_start()


check_start [](start = 34, length = 11)

rename to check_forwarding_resume?

lguohan · 2017-07-05T22:04:17Z

ansible/roles/test/files/ptftests/fast-reboot.py

-
-        self.assertTrue(no_routing_stop - no_routing_start < self.limit, "Downtime must be less then %s seconds" % self.test_params['fast_reboot_limit'])
+        self.log("Reboot time was %s" % str(no_routing_stop - self.reboot_start))
+        self.log("Number replies when control plane was down: %d Expected: %d" % (no_cp_replies, self.nr_vl_pkts))


what is this reply? can you explain?

lguohan · 2017-07-05T22:16:29Z

ansible/roles/test/files/ptftests/fast-reboot.py

+
+        return [('lo', lo_route, lo_valid, lo_prefix, lo_nexthop), ('vlan', vlan_route, vlan_valid, vlan_prefix, vlan_nexthop)]
+
+    def test_all_client_cases(self, output):


test_all_client_cases [](start = 8, length = 21)

rename to check_all_peer_status?

lguohan · 2017-07-05T22:22:57Z

ansible/roles/test/files/ptftests/fast-reboot.py

+            self.fails['dut'].add("Downtime must be less then %s seconds" % self.test_params['fast_reboot_limit'])
+        if no_routing_stop - self.reboot_start > datetime.timedelta(seconds=self.test_params['graceful_limit']):
+            self.fails['dut'].add("Fast-reboot cycle must be less then graceful limit %s seconds" % self.test_params['graceful_limit'])
+        if no_cp_replies < 0.95 * self.nr_vl_pkts:


if no_cp_replies < 0.95 * self.nr_vl_pkts: [](start = 5, length = 45)

why 0.95? when control plane is down, the data plane should be 100% up in our case, right?

In case fanout switches, Linux kernel, NIC drivers, ptf itself don't lose any packets which is not always true. So I made it 95% which work for me.

lguohan · 2017-07-05T22:26:37Z

ansible/roles/test/files/ptftests/fast-reboot.py

+      if len(non_zero) > 1 and non_zero[-1] < non_zero[-2]:
+          return non_zero[-2]
+      else:
+          return non_zero[-1]


maybe it is more robust just to find out the 90% of the data for non zero. Meaning 90% percentage of time, the you have the reachability to the n host, and the n should be equal to nr_vl_pkts.

Honestly It sounds more complicated to me.

lguohan · 2017-07-05T22:29:50Z

ansible/roles/test/files/ptftests/fast-reboot.py

        vlan_ip_range = self.test_params['vlan_ip_range']

        _, mask = vlan_ip_range.split('/')
        n_hosts = min(2**(32 - int(mask)) - 3, self.max_nr_vl_pkts)


n_hosts = min(2**(32 - int(mask)) - 3, self.max_nr_vl_pkts) [](start = 8, length = 59)

i think it is better to simply the test, and we just define the number of hosts under the vlan and send 1 packet per host every iteration.

sending multiple packet per host does not seem neccessary and it complicates the data interpretation later.

My code calculates the minimum number of hosts which could be used with current size of Vlan network. Our preference is to check against 1000 vlan hosts. But if size of vlan network is less then /22, we can't test for 1000 vlan hosts. So I take maximum possible value for the vlan network size..
Also. I send only one packet per host.

lguohan · 2017-07-05T22:32:46Z

ansible/roles/test/files/ptftests/fast-reboot.py

+
+            gr_active, timer = other['bgp_neig']
+            # wnen it's False, it's ok, wnen it's True, check that inactivity timer not less then 15 seconds
+            if gr_active and datetime.datetime.strptime(timer, '%H:%M:%S') < datetime.datetime(1900, 1, 1, second = 15):


15 [](start = 116, length = 2)

make this as an parameter?

lguohan · 2017-07-05T22:43:47Z

ansible/roles/test/files/ptftests/fast-reboot.py

+        if no_routing_stop - no_routing_start > self.limit:
+            self.fails['dut'].add("Downtime must be less then %s seconds" % self.test_params['fast_reboot_limit'])
+        if no_routing_stop - self.reboot_start > datetime.timedelta(seconds=self.test_params['graceful_limit']):
+            self.fails['dut'].add("Fast-reboot cycle must be less then graceful limit %s seconds" % self.test_params['graceful_limit'])


then [](start = 66, length = 4)

than

lguohan · 2017-07-05T22:47:48Z

ansible/roles/test/files/ptftests/fast-reboot.py

        self.log("ASIC was stopped, Waiting until it's up. Stop time: %s" % str(no_routing_start))
        self.timeout(self.task_timeout, "DUT hasn't started to work for %d seconds" % self.task_timeout)
-        no_routing_stop = self.check_start()
+        no_routing_stop, _ = self.check_start()


I wonder what should be the final exit criteria for fast-reboot? whether we need to check if all peer bgp session are up (exit from graceful restart mode)?

I defined the exit criteria as working dataplane. I can check that all bgp session are up at the end of the test.

lguohan

as comments

…y are down

* New version of fast reboot test

…lure (sonic-net#204)  ### Description of PR  Summary: Fixes # (issue) This PR fixes issue sonic-net#9052, where '_test_default_route_with_bgp_flap_' fails with "_Default route nexthops doesn't match the testbed topology_" issue. The count of the next hops is not matching with the upstream neighbor as per the topology (reading data from config facts). ### Type of change  - [x] Bug fix - [ ] Testbed and Framework(new/improvement) - [ ] Test case(new/improvement) ### Back port request - [ ] 202012 - [ ] 202205 - [ ] 202305 - [ ] 202311 - [x] 202405 ### Approach #### What is the motivation for this PR? - '_test_default_route_with_bgp_flap_' fails with "_Default route nexthops doesn't match the testbed topology_" issue. - The count of the next hops is not matching with the upstream neighbor as per the topology. #### How did you do it? - Instead of skipping T3 neighbors of AZNGHub, use the address family and next hops information to find out the list of upstream neighbors. #### How did you verify/test it? - Ran the above-mentioned test case on a T2 chassis and made sure the test passed without any issues. #### Any platform specific information? #### Supported testbed topology if it's a new test case? ### Documentation  ![image](https://github.com/user-attachments/assets/03516f51-63f0-41bb-be9e-eb9ae2c18d92) ![image](https://github.com/user-attachments/assets/0932f1fb-0754-4949-b50d-33a8007d878d)

* 664f0e2 2021-07-14 | [xrcvd]: Removed undefined symbol 'sfp_status_helper' (sonic-net#204) (HEAD, origin/202012) [Prince George] * 1b2d016 2021-06-16 | [CI] sonic-config-engine now depends on SONiC YANG packages (sonic-net#194) [Joe LeVeque] * 1cf5996 2021-07-14 | Introduce mgmtinit delay after transceiver module insertion (sonic-net#201) [Prince George] Signed-off-by: Guohan Lu <[email protected]>

… submodule head (sonic-net#12494) utilities: * 02eb899e 2022-07-12 | [config/load_mgmt_config] Support load IPv6 mgmt IP (sonic-net#2206) (sonic-net#2246) (sonic-net#2256) (HEAD -> 201811, github/201811) [Jing Kan] platform-daemon: * fc288cc 2022-05-05 | Mem leak caused by Xcvrd in Send-Q of REDIS-DB socket connection (sonic-net#260) (HEAD -> 201811, github/201811) [Prince George] platform-common: * edb062b 2022-02-09 | [sonic_sfp] Interpret sff 'int' element =0 as valid value (sonic-net#261) (HEAD -> 201811, github/201811) [Prince George] kernel: * 9d2d1a1 2022-02-11 | [201811] Increase log_buf_len size to 1M (HEAD -> 201811, github/201811) [Sujin Kang] * b34a213 2022-02-10 | [201811] Increase log_buf_len size to 1M [Samuel Angebault] * c4684cb 2022-02-11 | [201811] Apply kernel patches to fix emmc unreliability [Sujin Kang] * df68771 2022-02-03 | [201811] Apply kernel patches to fix emmc unreliability [Samuel Angebault] * f21cb06 2021-03-26 | [ci]: remove 201811 suffix (sonic-net#204) [lguohan] * 5439b2a 2021-02-08 | [dni_dps460] Add attributes to retrieve PMBus status command codes (sonic-net#194) [Arun Saravanan Balachandran] Signed-off-by: Ying Xie <[email protected]> Signed-off-by: Ying Xie <[email protected]>

Included commits in sonic-py-swsssdk ``` 63c75c1 2021-03-14 | Workaround Mellanox default vlan has no SAI_VLAN_ATTR_VLAN_ID attribute (sonic-net#103) [Qi Luo] ``` Included commits in sonic-snmpagent ``` a8c6e36 2021-03-15 | Implement rfc4363 FdbUpdater for lag inside vlan (sonic-net#204) [Qi Luo] ```

New version of fast reboot test

881ce89

msftclas added the cla-not-required label Jun 30, 2017

pavel-shirshov added 3 commits June 30, 2017 11:34

Update example of fast-reboot test runner

b3478d0

Use templates instead of two supervisor.conf files

b64f4c5

Output fails in error output

d4e05dc

pavel-shirshov added the Enhancement label Jun 30, 2017

Update example of fast-reboot test runner

882602e

lguohan reviewed Jul 5, 2017

View reviewed changes

lguohan requested changes Jul 5, 2017

View reviewed changes

pavel-shirshov added 2 commits July 6, 2017 13:59

Address comments

b4e04fc

Fix bug introduced with min_bgp_gr_timeout parameter

6f3c441

lguohan approved these changes Jul 7, 2017

View reviewed changes

pavel-shirshov added 3 commits July 24, 2017 17:09

LACP down and bgp routes down are expected. Measure time how long the…

3c2b75c

…y are down

Update documentation

b135a96

Adjust the test for using with linkstate helper support

bba56d2

Disable debug for Arista

a592da5

pavel-shirshov merged commit c88c151 into sonic-net:master Jul 27, 2017

pavel-shirshov deleted the pavelsh/fr_update branch July 27, 2017 21:56

lguohan pushed a commit that referenced this pull request Aug 3, 2017

[fast-reboot]: New version of fast reboot test (#204)

6cb6a52

* New version of fast reboot test


		return stdout, stderr, return_code

		def ssh_job(self, ip, queue):


		return [('lo', lo_route, lo_valid, lo_prefix, lo_nexthop), ('vlan', vlan_route, vlan_valid, vlan_prefix, vlan_nexthop)]

		def test_all_client_cases(self, output):

Conversation

pavel-shirshov commented Jun 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lguohan Jul 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pavel-shirshov Jul 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lguohan left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pavel-shirshov commented Jun 30, 2017 •

edited

Loading

lguohan Jul 5, 2017 •

edited

Loading

pavel-shirshov Jul 6, 2017 •

edited

Loading