[library/action_sync.py]: This module will make sure that actions such as add-topo, deploy-mg and remove-topo are in sync with current topo.

[praveen-li]

Description of PR

Summary:
Fixes # (issue)

[library/action_sync.py]: This module will make sure that actions such as add-topo, deploy-mg and remove-topo are in sync with current topo.

Changes:
-- New ansible module to make sure add-topo, deploy-mg and remove-topo are in sync.
-- related changes in other playbooks.

How I did:
-- Use shelve to store a DB in a file.
-- DB will have info about current topo, dut and server.
-- This will make sure, a topo is removed before adding new one.
-- Also deploy-mg, remove-topo will be allowed only for currently loaded topo.

Note: I did it for add-topo, deploy-mg and remove-topo , but can be extended for renumber etc.

Signed-off-by: Praveen Chaudhary pchaudhary@linkedin.com

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Approach

How I did:
-- Use shelve to store a DB in a file.
-- DB will have info about current topo, dut and server.
-- This will make sure, a topo is removed before adding new one.
-- Also deploy-mg, remove-topo will be allowed only for currently loaded topo.

What is the motivation for this PR?

Today from same sonic-mgmt docker, 2 users may end up running 2 separate add-topo commands which may messed up the Setup. Ideally deploy-mg, remove-topo should be allowed only for currently loaded topo. And add-topo must be allowed only after removing current-topo.
With this PR I use a shelve DB, which makes sure actions are in sync. Also it stored last executed commands.

How did you do it?

How I did:
-- Use shelve to store a DB in a file.
-- DB will have info about current topo, dut and server.
-- This will make sure, a topo is removed before adding new one.
-- Also deploy-mg, remove-topo will be allowed only for currently loaded topo.

How did you verify/test it?

Testing is recorded in a video, But below I copied

Steps:
1.) $ ./testbed-cli.sh add-topo vms-celdx10-t1-16 ~/.password -e ptf_imagetag=202006 -vvv

2.) $ ./testbed-cli.sh add-topo vms-celdx10-t1-8 ~/.password -e ptf_imagetag=202006 -vvv

If fails due to prev topo:

                    "2020-09-29 18:26:18.426128: -SUCC- add-topo topo=vms-celdx10-t1-16 dut=falco-test-dut04 server=server_2",
                    "2020-09-29 18:30:12.101652: -SUCC- run-tests topo=vms-celdx10-t1-16 dut=falco-test-dut04 server=server_2 test=bgp_facts"
,
                    "2020-09-29 18:30:41.133856: -SUCC- run-tests topo=vms-celdx10-t1-16 dut=falco-test-dut04 server=server_2 test=bgp_fact",
                    "2020-09-29 22:04:59.211082: -FAIL- add-topo topo=vms-celdx10-t1-8 dut=falco-open19-flex server=server_2"
                ],
                "success": [
                    "2020-09-29 17:09:04.402483: run-tests topo=vms-celdx10-t1-16 dut=falco-test-dut04 server=server_2 test=bgp_facts",
                    "2020-09-29 17:11:13.223194: remove-topo topo=vms-celdx10-t1-16 dut=falco-test-dut04 server=server_2",
                    "2020-09-29 18:26:18.425939: add-topo topo=vms-celdx10-t1-16 dut=falco-test-dut04 server=server_2",
                    "2020-09-29 18:30:12.101459: run-tests topo=vms-celdx10-t1-16 dut=falco-test-dut04 server=server_2 test=bgp_facts",
                    "2020-09-29 18:30:41.133659: run-tests topo=vms-celdx10-t1-16 dut=falco-test-dut04 server=server_2 test=bgp_fact"
                ]
            },
            "current_topo": {
                "deploy-mg": false,
                "dut": "falco-test-dut04",
                "server": "server_2",
                "topo_name": "vms-celdx10-t1-16"
            }

./testbed-cli.sh deploy-mg vms-celdx10-t0-8 lab ~/.password
FAIL

                    "2020-09-29 22:04:59.211082: -FAIL- deploy-mg topo=vms-celdx10-t1-8 dut=falco-open19-flex server=server_2"
                ],
                "success": [ .... ]
            },
            "current_topo": {
                "deploy-mg": false,
                "dut": "falco-test-dut04",
                "server": "server_2",
                "topo_name": "vms-celdx10-t1-16"
            }

3.) ./testbed-cli.sh deploy-mg vms-celdx10-t0-16 lab ~/.password

PASS

            "current_topo": {
                "deploy-mg": true,
                "dut": "falco-test-dut04",
                "server": "server_2",
                "topo_name": "vms-celdx10-t1-16"
            }

4.) ./testbed-cli.sh run-tests vms-celdx10-t0-16 lab bgp_facts
PASS
5.) ./testbed-cli.sh remove-topo vms-celdx10-t1-8 ~/.password -vvv

           "2020-09-29 22:09:47.984047: -FAIL- remove-topo topo=vms-celdx10-t1-8 dut=falco-open19-flex server=server_2"
            ],
            "success": [.....]
        },
        "current_topo": {
            "deploy-mg": false,
            "dut": "falco-test-dut04",
            "server": "server_2",
            "topo_name": "vms-celdx10-t1-16"
        }

6.) ./testbed-cli.sh remove-topo vms-celdx10-t1-16 ~/.password -vvv
PASS

#### Any platform specific information?

#### Supported testbed topology if it's a new test case?

### Documentation 
<!--
(If it's a new feature, new test case)
Did you update documentation/Wiki relevant to your implementation?
Link to the wiki page?
-->

[wangxin]

It is a good idea to have a method to track what has been executed on each testbed. But it seems that the current approach does not meet the needs of a lab with multiple different test setups.

[wangxin]

Assume we have multiple testbeds defined in testbed.csv: tb01, tb02.
If add-topo is executed for tb01, then run add-topo for tb02 would fail, right? Because db['current_topo'] is not none and db['current_topo']['topo_name'] would be tb01. The else branch will be hit.
If my understanding is correct, this approach won't work for a lab with multiple different test setups.