[fast-reboot] Stop services after killing containers to prevent automatic restart

[jleveque]

- What I did

Explicitly stop corresponding services after killing Docker containers to prevent the service from automatically restarting the container during fast-/warm-reboot.

We are beginning to configure the services which control Docker containers to automatically restart the service and all dependent services if the container should ever stop. In the case of fast-/warm-reboot, stopping the service could require up to 10 seconds to stop the container (due to the behavior of the underlying docker stop command). This could take too long, so in this script we call docker kill, which will work faster. However, after killing the container, the service which controls the container (once configured to restart) will restart the container after the specified interval, but we do not want the behavior under these fast-/warm-reboot circumstances. The services will be restarted after completing the reboot. Therefore, we stop the service immediately after killing the corresponding container to ensure the service cannot automatically restart the container.

[lguohan]

I can see similiar commands in sonic_installer, can you check them as well?

        run_command("docker exec -i bgp pkill -9 zebra")
        run_command("docker exec -i bgp pkill -9 bgpd")

[lguohan]

as comments

[jleveque]

sonic_installer does have calls to pkill -9 ..., but no calls to docker kill ..., which is what this PR addresses. For the calls to pkill -9 ..., we will need to address these when adding the auto-restart mechanism to these containers.

[pavel-shirshov]

what if the service is going to be restarted before it was stopped?
With this patch we introduce the race condition here.

[qiluo-msft]

Retest this please

[jleveque]

@pavel-shirshov: If someone or something had called systemctl restart ... on one of these services and this script called systemctl stop ... while the restart operation was still in progress, whichever command was run last (in this case, systemctl stop ...) would override the pending systemctl restart .... Do you have some other concern?

[pavel-shirshov]

@jleveque
Current behavior is following:

1. We kill a process/processes inside of the docker container
2. We stop the container itself because we afraid the process would be restarted by supervisorctl inside of the container (fix me if I'm wrong here)

So I think if we don't fast enough to stop the container, but supervisorctl restarted just killed processes already?

[jleveque]

These processes are not set to be restarted by supervisor inside the containers. The restart mechanism uses a supervisor event listener to listen for processes exiting, which then causes supervisor to exit and subsequently the container itself. Then, systemd will restart the entire container. Here we're killing the container, then stopping the systemd service to prevent it from restarting the container.

[jleveque]

Retest this please

[jleveque]

Retest this please

[pavel-shirshov]

@jleveque Got it
Thanks