Skip to content

Avoid running command in exited ptf docker container#10286

Merged
wangxin merged 1 commit intosonic-net:masterfrom
wangxin:fix-kill-ptf-exabgp
Oct 11, 2023
Merged

Avoid running command in exited ptf docker container#10286
wangxin merged 1 commit intosonic-net:masterfrom
wangxin:fix-kill-ptf-exabgp

Conversation

@wangxin
Copy link
Copy Markdown
Collaborator

@wangxin wangxin commented Oct 11, 2023

Description of PR

Summary:
Fixes # (issue)

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Back port request

  • 201911
  • 202012
  • 202205

Approach

What is the motivation for this PR?

While stopping PTF container, "ptf_control" module is executed to kill all processes in the PTF container.
The original code checks if the PTF container's Pid exists before running command in the PTF container. Unfortunately, this check is not enough. PTF docker container in exited status still has Pid.

How did you do it?

This change improved the code for getting PTF container's Pid. When PTF container is not in "running" status, always return None for PTF container's Pid.

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

@wangxin
Avoid running command in exited ptf docker container
ff40911 
While stopping PTF container, "ptf_control" module is executed to
kill all processes in the PTF container.
The original code checks if the PTF container's Pid exists before
running command in the PTF container. Unfortunately, this check
is not enough. PTF docker container in exited status still has Pid.

This change improved the code for getting PTF container's Pid.
When PTF container is not in "running" status, always return None
for PTF container's Pid.

Signed-off-by: Xin Wang <[email protected]>
@wangxin wangxin requested a review from yejianquan October 11, 2023 03:34
yejianquan
yejianquan approved these changes Oct 11, 2023
View reviewed changes
Copy link
Copy Markdown
Collaborator

@yejianquan yejianquan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangxin wangxin merged commit 2192c2f into sonic-net:master Oct 11, 2023
@mssonicbld
Copy link
Copy Markdown
Collaborator

mssonicbld commented Oct 30, 2023

@wangxin PR conflicts with 202305 branch

wangxin added a commit to wangxin/sonic-mgmt that referenced this pull request Oct 30, 2023
@wangxin
Avoid running command in exited ptf docker container (sonic-net#10286)
48d7316 
While stopping PTF container, "ptf_control" module is executed to
kill all processes in the PTF container.
The original code checks if the PTF container's Pid exists before
running command in the PTF container. Unfortunately, this check
is not enough. PTF docker container in exited status still has Pid.

This change improved the code for getting PTF container's Pid.
When PTF container is not in "running" status, always return None
for PTF container's Pid.

Signed-off-by: Xin Wang <[email protected]>
@wangxin wangxin mentioned this pull request Oct 30, 2023
Merged
7 tasks
@mssonicbld
Copy link
Copy Markdown
Collaborator

mssonicbld commented Oct 30, 2023

Cherry-pick PR to 202305: #10524

@mssonicbld mssonicbld mentioned this pull request Oct 30, 2023
Merged
6 tasks
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Oct 30, 2023
@wangxin @mssonicbld
Avoid running command in exited ptf docker container (sonic-net#10286)
f1b682d 
While stopping PTF container, "ptf_control" module is executed to
kill all processes in the PTF container.
The original code checks if the PTF container's Pid exists before
running command in the PTF container. Unfortunately, this check
is not enough. PTF docker container in exited status still has Pid.

This change improved the code for getting PTF container's Pid.
When PTF container is not in "running" status, always return None
for PTF container's Pid.

Signed-off-by: Xin Wang <[email protected]>
wangxin added a commit that referenced this pull request Oct 30, 2023
@wangxin
[202205] Improve the cleanup of PTF container (#10523)
13c6b12 
Cherry-pick #10069 and #10286 to 202205 branch.

* Improve the cleanup of processes and interfaces before stopping PTF container (#10244 

What is the motivation for this PR?
We still observe issues with "testbed-cli.sh remove-topo" and "testbed-cli.sh restart-ptf":

Server may crash and run into CPU softlock issue.
Some exabgp process cannot be fully stopped and "restart-ptf" may fail.
The expectation is that remove-topo and restart-ptf can always be successful. And of course, no server crash.
Possible reason of server crash:

Some exabgp processes are still running in PTF container while we remove the container. This could cause server crash.
Some network interfaces are in the PTF container's network namespace while we remove the container.

How did you do it?
Added a customized module "ptf_control" to stop&kill processes running in PTF container in a more aggressive and reliable way.
Improve the vm_topology module to remove network interfaces from the PTF container in the "unbind" procedure.
Added a vm_topology "unbind" step in the "testbed-cli.sh restart-ptf" procedure.
Updated some "ip link" commands to fully compliant with the syntax in "ip link help".

How did you verify/test it?
Tested the add-topo/remove-topo on both physical and KVM testbed.
Tested restart-ptf on phsycial testbed.

* Avoid running command in exited ptf docker container (#10286)

While stopping PTF container, "ptf_control" module is executed to
kill all processes in the PTF container.
The original code checks if the PTF container's Pid exists before
running command in the PTF container. Unfortunately, this check
is not enough. PTF docker container in exited status still has Pid.

This change improved the code for getting PTF container's Pid.
When PTF container is not in "running" status, always return None
for PTF container's Pid.

Signed-off-by: Xin Wang <[email protected]>
mssonicbld pushed a commit that referenced this pull request Oct 30, 2023
@wangxin @mssonicbld
Avoid running command in exited ptf docker container (#10286)
a9334f3 
While stopping PTF container, "ptf_control" module is executed to
kill all processes in the PTF container.
The original code checks if the PTF container's Pid exists before
running command in the PTF container. Unfortunately, this check
is not enough. PTF docker container in exited status still has Pid.

This change improved the code for getting PTF container's Pid.
When PTF container is not in "running" status, always return None
for PTF container's Pid.

Signed-off-by: Xin Wang <[email protected]>
wangxin added a commit to wangxin/sonic-mgmt that referenced this pull request Oct 31, 2023
@wangxin
Avoid running command in exited ptf docker container (sonic-net#10286)
b686748 
While stopping PTF container, "ptf_control" module is executed to
kill all processes in the PTF container.
The original code checks if the PTF container's Pid exists before
running command in the PTF container. Unfortunately, this check
is not enough. PTF docker container in exited status still has Pid.

This change improved the code for getting PTF container's Pid.
When PTF container is not in "running" status, always return None
for PTF container's Pid.

Signed-off-by: Xin Wang <[email protected]>
@wangxin wangxin mentioned this pull request Oct 31, 2023
Merged
7 tasks
yejianquan pushed a commit that referenced this pull request Nov 9, 2023
@wangxin
[202012] Improve the cleanup of PTF container (#10533)
4473f6d 
* Improve the cleanup of processes and interfaces before stopping PTF container (#10069)

What is the motivation for this PR?
We still observe issues with "testbed-cli.sh remove-topo" and "testbed-cli.sh restart-ptf":

Server may crash and run into CPU softlock issue.
Some exabgp process cannot be fully stopped and "restart-ptf" may fail.
The expectation is that remove-topo and restart-ptf can always be successful. And of course, no server crash.
Possible reason of server crash:

Some exabgp processes are still running in PTF container while we remove the container. This could cause server crash.
Some network interfaces are in the PTF container's network namespace while we remove the container.

How did you do it?
Added a customized module "ptf_control" to stop&kill processes running in PTF container in a more aggressive and reliable way.
Improve the vm_topology module to remove network interfaces from the PTF container in the "unbind" procedure.
Added a vm_topology "unbind" step in the "testbed-cli.sh restart-ptf" procedure.
Updated some "ip link" commands to fully compliant with the syntax in "ip link help".

How did you verify/test it?
Tested the add-topo/remove-topo on both physical and KVM testbed.
Tested restart-ptf on phsycial testbed.

* Avoid running command in exited ptf docker container (#10286)

While stopping PTF container, "ptf_control" module is executed to
kill all processes in the PTF container.
The original code checks if the PTF container's Pid exists before
running command in the PTF container. Unfortunately, this check
is not enough. PTF docker container in exited status still has Pid.

This change improved the code for getting PTF container's Pid.
When PTF container is not in "running" status, always return None
for PTF container's Pid.

Signed-off-by: Xin Wang <[email protected]>

* Install right version of docker.py for ubuntu22.04

---------

Signed-off-by: Xin Wang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yejianquan yejianquan yejianquan approved these changes

Assignees

No one assigned

Labels

Approved for 202305 branch Included in 202012 branch Included in 202205 branch Included in 202305 branch Request for 202012 branch Request for 202205 branch Request for 202305 branch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wangxin @mssonicbld @yejianquan