diff --git a/ansible/README.test.md b/ansible/README.test.md index 8db0c21949c..2d434d67103 100644 --- a/ansible/README.test.md +++ b/ansible/README.test.md @@ -12,170 +12,289 @@ ### **Run test by test case name** -This is going to be the supported method to call individual test case going forward. All test cases name and calling variables and applied topologies are specified in [ansible/roles/test/vars/testcases.yml](roles/test/vars/testcases.yml) +All test cases name and calling variables and applied topologies are specified in [ansible/roles/test/vars/testcases.yml](roles/test/vars/testcases.yml) -When calling test, testbed_name is the main entry to pickup/understand the testbed information associated with the test (ie. ptf_host and tyestbed_type, VMs info...). testbed_name is inherited from your own `ansible/testbed.csv` file. The first column of each line(one testbed topology definition) is the unique name of the testbed and will be used in testbed_name option when calling test. +When calling test, testbed_name is the main entry to pickup/understand the testbed information associated with the test (ie. ptf_host and testbed_type, VMs info...). testbed_name is inherited from your own `ansible/testbed.csv` file. The first column of each line(one testbed topology definition) is the unique name of the testbed and will be used in testbed_name option when calling test. ***Example of running a test case:*** - `ansible-playbook -i lab -l str-s6000-acs-1 test_sonic.yml -e testbed_name={TESTBED_NAME} -e testcase_name={TESTCASE_NAME}` + `ansible-playbook -i {INVENTORY} -l {DUT_NAME} test_sonic.yml -e testbed_name={TESTBED_NAME} -e testcase_name={TESTCASE_NAME}` Where: `testcase_name=bgp_fact` `testbed_name=vms-t1-lag` ---- -### **Run test by test case tag `(DEPRECATING)`** - -When Ansible running playbooks by tag, it first include all tasks(all test cases) within test role shich not relate to specific tag. It's very slow along with adding more test cases and it occupied too much resource other than just run one test case using specific tag. It does not scale. - -We newly added a run test case by test name option(see above setion). Running test by tag option won’t be actively maintained going forward, but will backward compatible for all already working test cases, and eventually will be phaseout. There still going to be more improvement after the initial check in. - +- Replace {INVENTORY} in each command line with the inventory file name - Replace {DUT_NAME} in each command line with the host name of switch under test -- Replace {PTF_HOST} in each command line with the host name or IP of the PTF testbed host -- Replace {TESTBED_TYPE} in each command line with the type of the testbed being used +- Replace {TESTBED_NAME} in each command line with the first column in your 'ansible/testbed.csv' file associated with the DUT_NAME +- Replace {TESTCASE_NAME} in each command line with the testcase tag from 'roles/test/vars/testcases.yml' +--- ##### ACL tests ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --tags acl --extra-vars "testbed_type={TESTBED_TYPE} ptf_host={PTF_HOST}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=acl -e testbed_name={TESTBED_NAME} ``` -- Requires switch connected to a t1 or t1-lag testbed +- Requires switch connected to a VM set testbed ##### ARP tests ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags arp --extra-vars "ptf_host={PTF_HOST}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=arp -e testbed_name={TESTBED_NAME} ``` - Requires switch connected to a PTF testbed ##### BGP facts verification test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags bgp_fact +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=bgp_fact -e testbed_name={TESTBED_NAME} ``` - Requires switch connected to a VM set testbed +##### BGP Multipath Relax test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=bgp_multipath_relax -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a VM set testbed +- This test only works for T1 related topologies(t1, t1-lag, ...) +- You might need to redeploy your VMs before you run this test due to the change for ToR VM router configuration changes + `./testbed-cli.sh config-vm your-topo-name(vms1-1) your-vm-name(VM0108)` will do this for you + +##### Config test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=config -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a VM set testbed + +##### Continuous Reboot test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=continuous_reboot -e testbed_name={TESTBED_NAME} -e repeat_count={REPEAT_COUNT} +``` +- Requires switch connected to a VM set testbed +- Replace {REPEAT_COUNT} with the number of times the reboot has to be done. Default: 3 + ##### CoPP test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags copp --extra-vars "ptf_host={PTF_HOST}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=copp -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a PTF testbed + +##### CRM test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=crm -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a VM set testbed + +##### DECAP test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=decap -e testbed_name={TESTBED_NAME} ``` - Requires switch connected to a PTF testbed ##### DHCP relay test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags dhcp_relay --extra-vars "ptf_host={PTF_HOST}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=dhcp_relay -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a PTF testbed + +##### DIP SIP test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=dip_sip -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a PTF testbed + +##### DIR BCAST test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=dir_bcast -e testbed_name={TESTBED_NAME} ``` - Requires switch connected to a PTF testbed ##### ECN WRED test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags ecn_wred +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=ecn_wred -e testbed_name={TESTBED_NAME} ``` - Requires switch connected to a VM testbed ##### Everflow_testbed test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --tags everflow_testbed --extra-vars "testbed_type={TESTBED_TYPE} ptf_host={PTF_HOST}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=everflow_testbed -e testbed_name={TESTBED_NAME} ``` -- Requires switch connected to a VM testbed +- Requires switch connected to a PTF testbed + +##### Fast reboot link util test +``` +ansible-playbook -i linkstate/testbed_inv.py -e target_host={TESTBED_NAME} linkstate/{STATE}.yml +``` +- Requires switch connected to a PTF testbed +- Replace {STATE} with up or down +- This test is run before running the fast-reboot/warm-reboot tests and is used to enable link state propagation from fanout to the VMS + +##### FAST REBOOT test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=fast-reboot -e testbed_name={TESTBED_NAME} -e new_sonic_image={IMAGE_URL} --stay_in_target_image={VALUE1} --cleanup_old_sonic_images={VALUE2} +``` +- Requires switch connected to PTF testbed +- Replace {IMAGE_URL} with the link pointing to the next image to fast-reboot into +- Replace {VALUE1} and {VALUE2} with true/false. Default: false +- stay_in_target_image parameter decides if the DUT should be reverted back to the old image after fast-reboot +- cleanup_old_sonic_images parameter will decide if all the images on the DUT should be cleaned up except for the current and the next images ##### FDB test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --tags fdb --extra-vars "testbed_type={TESTBED_TYPE} ptf_host={PTF_HOST} [ipv6=True]" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=fdb -e testbed_name={TESTBED_NAME} ``` -- Requires switch connected to a VM testbed(t0); default IPv4 +- Requires switch connected to a PTF testbed -##### FIB test +##### FDB Mac expire test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --tags fib --extra-vars "testbed_type={TESTBED_TYPE} ptf_host={PTF_HOST} [ipv4=Flase]" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=fdb_mac_expire -e testbed_name={TESTBED_NAME} ``` -- Requires switch connected to a VM testbed; default IPv4 +- Requires switch connected to a PTF testbed -##### MTU test +##### FIB v4 test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --tags mtu --extra-vars "testbed_type={TESTBED_TYPE} ptf_host={PTF_HOST}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=fib -e testbed_name={TESTBED_NAME} -e ipv6=False ``` -- Requires switch connected to a t1 or t1-lag testbed +- Requires switch connected to a PTF testbed -##### Fast-Reboot test +##### FIB v6 test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags fast_reboot --extra-vars "ptf_host={PTF_HOST}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=fib -e testbed_name={TESTBED_NAME} -e ipv4=False ``` - Requires switch connected to a PTF testbed -##### IPDecap Test +##### LAG test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} -tags decap --extra-vars "testbed_type={TESTBED_TYPE} ptf_host={PTF_HOST} dscp_mode=pipe|uniform" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=lag_2 -e testbed_name={TESTBED_NAME} ``` -- Require VM testbed -- dscp_mode=pipe: if your ASIC type is Broadcom; -- dscp_mode=uniform: if your ASIC type is Mellanox +- Requires switch connected to a VM set testbed with lags configured -##### Lag-2 test +##### Link Flap test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --tags lag-2 --extra-vars "testbed_type={TESTBED_TYPE} ptf_host={PTF_HOST}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=link_flap -e testbed_name={TESTBED_NAME} ``` -- Requires switch connected to a VM testbed with lag configured (t0, t1-lag) +- Requires switch connected to fanout switch. VM or PTF testbed ##### LLDP test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME},lldp_neighbors --become --tags lldp +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME},lldp_neighbors -e testcase_name=lldp -e testbed_name={TESTBED_NAME} ``` - Requires switch connected to a VM set testbed -##### Link flap test +##### MAC read test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME}, --become --tags link_flap +ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} -e testbed_name={TESTBED_NAME} -e testbed_type={TESTBED_TYPE} -e testcase_name=read_mac -e iterations={ITERATIONS} -e image1={IMAGE1} -e image2={IMAGE2} ``` -- Requires switch connected to fanout switch. VM or PTF testbed +- Replace {ITERATIONS} with the integer number of image flipping iterations. +- Replace {IMAGE1} and {IMAGE2} with URLs to the specific SONiC binary images. +- Requires switch connected to a VM set testbed + +##### Mem check test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=mem_check -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a VM set or PTF testbed + +##### MTU test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=mtu -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to PTF testbed + +##### Neighbor Mac test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=neighbor -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a PTF testbed + +##### Neighbor Mac address change test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=neighbor_mac_noptf -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a VM set or PTF testbed ##### NTP test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags ntp +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=ntp -e testbed_name={TESTBED_NAME} ``` +- Requires switch connected to a VM set or PTF testbed -##### SNMP tests +##### PFC watchdog test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=pfc_wd -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a PTF testbed + +##### Port Toggle test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags snmp,snmp_cpu,snmp_interfaces +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=port_toggle -e testbed_name={TESTBED_NAME} ``` -- Require to run Anisble-playbook from docker-sonic-mgmt container. +- Requires switch connected to a VM set or PTF testbed + +##### Reboot test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=reboot -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a VM set or PTF testbed ##### Sensors test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags sensors +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=sensors -e testbed_name={TESTBED_NAME} ``` +- Requires switch connected to a VM set or PTF testbed -##### Syslog test +##### Service ACL test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags syslog +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=service_acl -e testbed_name={TESTBED_NAME} ``` +- Requires switch connected to a VM set or PTF testbed -##### PFC WD test +##### SNMP tests ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --tags pfc_wd --extra-vars "testbed_type={TESTBED_TYPE}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=snmp -e testbed_name={TESTBED_NAME} ``` +- Requires switch connected to a VM set or PTF testbed -##### BGP multipath relax test +##### Syslog test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --tags bgp_multipath_relax --extra-vars "testbed_type={TESTBED_TYPE}" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=syslog -e testbed_name={TESTBED_NAME} ``` -This test only works for T1 related topologies(t1, t1-lag, ...) -You might need to redeploy your VMs before you run this test due to the change for ToR VM router configuration changes -`./testbed-cli.sh config-vm your-topo-name(vms1-1) your-vm-name(VM0108)` will do this for you +- Requires switch connected to a VM set or PTF testbed ##### VLAN test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} -e "testbed_name={TESTBED_NAME} testcase_name=vlan" +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=vlan -e testbed_name={TESTBED_NAME} ``` -- Requires switch connected to a t0 testbed +- Requires switch connected to a PTF testbed - Requires switch connected to fanout switch and fanout switch need support [QinQ](https://en.wikipedia.org/wiki/IEEE_802.1ad). -### CRM test +##### VNET test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} --become --tags crm +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=vnet_vxlan -e testbed_name={TESTBED_NAME} ``` +- Requires switch connected to a PTF testbed -##### MAC read test +##### Vxlan decap test ``` -ansible-playbook test_sonic.yml -i inventory --limit {DUT_NAME} -e testbed_name={TESTBED_NAME} -e testbed_type={TESTBED_TYPE} -e testcase_name=read_mac -e iterations={ITERATIONS} -e image1={IMAGE1} -e image2={IMAGE2} +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=vxlan-decap -e testbed_name={TESTBED_NAME} ``` -- Replace {ITERATIONS} with the integer number of image flipping iterations. -- Replace {IMAGE1} and {IMAGE2} with URLs to the specific SONiC binary images. +- Requires switch connected to a PTF testbed + +##### Warm reboot test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=warm-reboot -e testbed_name={TESTBED_NAME} -e new_sonic_image={IMAGE_URL} --stay_in_target_image={VALUE1} --cleanup_old_sonic_images={VALUE2} +``` +- Requires switch connected to PTF testbed +- Replace {IMAGE_URL} with the link pointing to the next image to warm-reboot into +- Replace {VALUE1} and {VALUE2} with true/false. Default: false +- stay_in_target_image parameter decides if the DUT should be reverted back to the old image after warm-reboot +- cleanup_old_sonic_images parameter will decide if all the images on the DUT should be cleaned up except for the current and the next images + +##### Warm reboot FIB test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=warm-reboot-fib -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a PTF testbed + +##### Warm reboot Sad test +``` +ansible-playbook test_sonic.yml -i {INVENTORY} --limit {DUT_NAME} -e testcase_name=warm-reboot-sad -e testbed_name={TESTBED_NAME} +``` +- Requires switch connected to a PTF testbed