[Snappi] Support for DUT statistics - Interface counters, PFC counters, and Queue counters. (sonic-net#13848)

Description of PR
Support to pull the DUT statistics, namely interface counters, PFC counters and Queue counters.

This support will enable users to pull the DUT statistics run-time and capture them in form of dictionary for further processing.

Summary:
Fixes # (issue)
sonic-net#13843

Approach
What is the motivation for this PR?
Currently, there is no one stop-place to fetch the DUT statistics and use them for verification, especially the SNAPPI testcases. There should be some way to pull the DUT statistics ( at least important ones) and use them for verification.

How did you do it?
Clear counters:
Common function to clear interface, PFC and queue counter stats.

Interface counters:
Ability to fetch the interface counters for a given DUT and port. Returns dictionary with keys - Rx and Tx packet count, Rx and Tx failures (Error + drops + ovr), and Rx and Tx throughput in Mbps.

PFC counters:
Ability to fetch the PFC counters for a given DUT and port. Returns dictionary with keys - Rx and Tx PFC for the given DUT, port and priority.

Queue counters:
Ability to fetch the Queue counters for a given DUT and port. Internally calls get_egress_queue_count for all the priorities. Returns dictionary with as key with transmitted packets as counter.

How did you verify/test it?
These functions are called as part of new testcases for PFC-ECN.

    for dut, port in dutport_list:
        f_stats = update_dict(m, f_stats, interface_stats(dut, port))
        f_stats = update_dict(m, f_stats, get_pfc_count(dut, port))
        f_stats = update_dict(m, f_stats, get_queue_count(dut, port))
These are then used to create CSV with raw DUT statistics and then to summarize the test in the end.

co-authorized by: [email protected]

Author: amitpawar12

tests/common/snappi_tests/common_helpers.py

@@ -18,6 +18,8 @@
 from tests.common.mellanox_data import is_mellanox_device as isMellanoxDevice
 from ipaddress import IPv6Network, IPv6Address
 from random import getrandbits
+from tests.common.portstat_utilities import parse_portstat
+from collections import defaultdict
 
 
 def increment_ip_address(ip, incr=1):
@@ -959,14 +961,22 @@ def get_egress_queue_count(duthost, port, priority):
     Returns:
         tuple (int, int): total count of packets and bytes in the queue
     """
-    raw_out = duthost.shell("show queue counters {} | sed -n '/UC{}/p'".format(port, priority))['stdout']
-    total_pkts = raw_out.split()[2] if 2 < len(raw_out.split()) else "0"
-    if total_pkts == "N/A":
-        total_pkts = "0"
+    # If DUT is multi-asic, asic will be used.
+    if duthost.is_multi_asic:
+        asic = duthost.get_port_asic_instance(port).get_asic_namespace()
+        raw_out = duthost.shell("sudo ip netns exec {} show queue counters {} | sed -n '/UC{}/p'".
+                                format(asic, port, priority))['stdout']
+        total_pkts = "0" if raw_out.split()[2] == "N/A" else raw_out.split()[2]
+        total_bytes = "0" if raw_out.split()[3] == "N/A" else raw_out.split()[3]
+    else:
+        raw_out = duthost.shell("show queue counters {} | sed -n '/UC{}/p'".format(port, priority))['stdout']
+        total_pkts = raw_out.split()[2] if 2 < len(raw_out.split()) else "0"
+        if total_pkts == "N/A":
+            total_pkts = "0"
 
-    total_bytes = raw_out.split()[3] if 3 < len(raw_out.split()) else "0"
-    if total_bytes == "N/A":
-        total_bytes = "0"
+        total_bytes = raw_out.split()[3] if 3 < len(raw_out.split()) else "0"
+        if total_bytes == "N/A":
+            total_bytes = "0"
 
     return int(total_pkts.replace(',', '')), int(total_bytes.replace(',', ''))
 
@@ -1061,3 +1071,120 @@ def start_pfcwd_fwd(duthost, asic_value=None):
         stop_pfcwd(duthost, asic_value)
         duthost.shell('sudo ip netns exec {} pfcwd start --action forward 200 --restoration-time 200'.
                       format(asic_value))
+
+
+def clear_counters(duthost, port):
+    """
+    Clear PFC, Queuecounters, Drop and generic counters from SONiC CLI.
+    Args:
+        duthost (Ansible host instance): Device under test
+        port (str): port name
+    Returns:
+        None
+    """
+
+    duthost.shell("sudo sonic-clear counters \n")
+    duthost.shell("sudo sonic-clear pfccounters \n")
+    duthost.shell("sudo sonic-clear priority-group drop counters \n")
+    duthost.shell("sonic-clear counters \n")
+    duthost.shell("sonic-clear pfccounters \n")
+
+    if (duthost.is_multi_asic):
+        asic = duthost.get_port_asic_instance(port).get_asic_namespace()
+        duthost.shell("sudo ip netns exec {} sonic-clear queuecounters \n".format(asic))
+        duthost.shell("sudo ip netns exec {} sonic-clear dropcounters \n".format(asic))
+    else:
+        duthost.shell("sonic-clear queuecounters \n")
+        duthost.shell("sonic-clear dropcounters \n")
+
+
+def get_interface_stats(duthost, port):
+    """
+    Get the Rx and Tx port failures, throughput and pkts from SONiC CLI.
+    This is the equivalent of the "show interface counters" command.
+    Args:
+        duthost (Ansible host instance): device under test
+        port (str): port name
+    Returns:
+        i_stats (dict): Returns various parameters for given DUT and port.
+    """
+    # Initializing nested dictionary i_stats
+    i_stats = defaultdict(dict)
+    i_stats[duthost.hostname][port] = {}
+
+    n_out = parse_portstat(duthost.command('portstat -i {}'.format(port))['stdout_lines'])[port]
+    # rx_err, rx_ovr and rx_drp are counted in single counter rx_fail
+    # tx_err, tx_ovr and tx_drp are counted in single counter tx_fail
+    rx_err = ['rx_err', 'rx_ovr', 'rx_drp']
+    tx_err = ['tx_err', 'tx_ovr', 'tx_drp']
+    rx_fail = 0
+    tx_fail = 0
+    for m in rx_err:
+        rx_fail = rx_fail + int(n_out[m].replace(',', ''))
+    for m in tx_err:
+        tx_fail = tx_fail + int(n_out[m].replace(',', ''))
+
+    # Any throughput below 1MBps is measured as 0 for simplicity.
+    thrput = n_out['rx_bps']
+    if thrput.split(' ')[1] == 'MB/s' and (thrput.split(' ')[0]) != '0.00':
+        i_stats[duthost.hostname][port]['rx_thrput_Mbps'] = float(thrput.split(' ')[0]) * 8
+    else:
+        i_stats[duthost.hostname][port]['rx_thrput_Mbps'] = 0
+    thrput = n_out['tx_bps']
+    if thrput.split(' ')[1] == 'MB/s' and (thrput.split(' ')[0]) != '0.00':
+        i_stats[duthost.hostname][port]['tx_thrput_Mbps'] = float(thrput.split(' ')[0]) * 8
+    else:
+        i_stats[duthost.hostname][port]['rx_thrput_Mbps'] = 0
+
+    i_stats[duthost.hostname][port]['rx_pkts'] = int(n_out['rx_ok'].replace(',', ''))
+    i_stats[duthost.hostname][port]['tx_pkts'] = int(n_out['tx_ok'].replace(',', ''))
+    i_stats[duthost.hostname][port]['rx_fail'] = rx_fail
+    i_stats[duthost.hostname][port]['tx_fail'] = tx_fail
+
+    return i_stats
+
+
+def get_queue_count_all_prio(duthost, port):
+    """
+    Get the egress queue count in packets and bytes for a given port and priority from SONiC CLI.
+    This is the equivalent of the "show queue counters" command.
+    Args:
+        duthost (Ansible host instance): device under test
+        port (str): port name
+    Returns:
+        queue_dict (dict): key-value with key=dut+port+prio and value=queue count
+    """
+    # Initializing nested dictionary queue_dict
+    queue_dict = defaultdict(dict)
+    queue_dict[duthost.hostname][port] = {}
+
+    # Preparing the dictionary for all 7 priority queues.
+    for priority in range(7):
+        total_pkts, _ = get_egress_queue_count(duthost, port, priority)
+        queue_dict[duthost.hostname][port]['prio_' + str(priority)] = total_pkts
+
+    return queue_dict
+
+
+def get_pfc_count(duthost, port):
+    """
+    Get the PFC frame count for a given port from SONiC CLI
+    Args:
+        duthost (Ansible host instance): device under test
+        port (str): port name
+    Returns:
+        pfc_dict (dict) : Returns Rx and Tx PFC for the given DUT and interface.
+    """
+    pfc_dict = defaultdict(dict)
+    pfc_dict[duthost.hostname][port] = {}
+    raw_out = duthost.shell("show pfc counters | sed -n '/Port Tx/,/^$/p' | grep '{} '".format(port))['stdout']
+    pause_frame_count = raw_out.split()
+    for m in range(1, len(pause_frame_count)):
+        pfc_dict[duthost.hostname][port]['tx_pfc_'+str(m-1)] = int(pause_frame_count[m].replace(',', ''))
+
+    raw_out = duthost.shell("show pfc counters | sed -n '/Port Rx/,/^$/p' | grep '{} '".format(port))['stdout']
+    pause_frame_count = raw_out.split()
+    for m in range(1, len(pause_frame_count)):
+        pfc_dict[duthost.hostname][port]['rx_pfc_'+str(m-1)] = int(pause_frame_count[m].replace(',', ''))
+
+    return pfc_dict