[Services] Restart database service upon unexpected critical process exit.

[yozhao101]

Signed-off-by: Yong Zhao yozhao@microsoft.com

• What I did
  Restart database service if one of critical processes running in database container exited or crashed abnormally.

• How I did it
  Generally I follow the framework created by Joe to implement this feature in database container.
  First, add supervisor-proc-exit-listener event listener option in Supervisord configuration file in database docker container. Supervisord will read a list of critical processes for which to monitor the unexpected crashed and exited.
  Second, configure database.service to always auto-restart the service if it stops, with a delay of 30 seconds. Also set a rate limit of 3 restarts within 20 minutes (1200 seconds).

• How to verify it
  On your switch device, please use docker ps command to list all running docker containers.
  Then use docker exec -it container_id bash to login target container. Typing top command
  on the shell will display all the processes dynamically and you will spot the process id of one
  of the critical processes. Finally type the command kill -9 process_id to terminate one process.
  After exiting the container, you can use watch -n 1 docker ps to dynamically see the restart
  of database container.