Improve the cleanup of processes and interfaces before stopping PTF container

[wangxin]

Description of PR

Summary:
Fixes # (issue)

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 201911
• 202012
• 202205

Approach

What is the motivation for this PR?

We still observe issues with "testbed-cli.sh remove-topo" and "testbed-cli.sh restart-ptf":

1. Server may crash and run into CPU softlock issue.
2. Some exabgp process cannot be fully stopped and "restart-ptf" may fail.
   The expectation is that remove-topo and restart-ptf can always be successful. And of course, no server crash.

Possible reason of server crash:

1. Some exabgp processes are still running in PTF container while we remove the container. This could cause server crash.
2. Some network interfaces are in the PTF container's network namespace while we remove the container.

How did you do it?

1. Added a customized module "ptf_control" to stop&kill processes running in PTF container in a more aggressive and reliable way.
2. Improve the vm_topology module to remove network interfaces from the PTF container in the "unbind" procedure.
3. Added a vm_topology "unbind" step in the "testbed-cli.sh restart-ptf" procedure.
4. Updated some "ip link" commands to fully compliant with the syntax in "ip link help".

How did you verify/test it?

Tested the add-topo/remove-topo on both physical and KVM testbed.
Tested restart-ptf on physical testbed.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

ansible/roles/vm_set/library/ptf_control.py:113:1: E302 expected 2 blank lines, found 1

flake8...............................................(no files to check)Skipped
check conditional mark sort..............................................Passed

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
   the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
   docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

ansible/roles/vm_set/library/ptf_control.py:75:121: E501 line too long (123 > 120 characters)

flake8...............................................(no files to check)Skipped
check conditional mark sort..............................................Passed

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
   the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
   docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[yejianquan]

LGTM

[lizhijianrd]

What is this command used for? (I have same confusion at line 108)

[wangxin]

It is for debugging purpose. By default, this module can generate logs like /tmp/ptf_control_xxx.log.
With this command, we can clearly see the processes running in PTF docker before and after the killing. Next time if server crash happens again, we may be able to get some clue from the log.

Recently we noticed some tests may create exabgp processes like "exabgp-psudoswitch1" in PTF container. So, a more aggressive way is required to kill the processes. Also, it would be better to collect more information for debugging if the issue happens again in the future.

[mssonicbld]

@wangxin PR conflicts with 202205 branch

[mssonicbld]

@wangxin PR conflicts with 202012 branch

[mssonicbld]

Cherry-pick PR to 202305: #10521