[services] Restart SwSS service upon unexpected critical process exit

[jleveque]

- What I did
Restart SwSS service (and also restart dependent services) if any critical processes running in the swss container exit abnormally.

- How I did it

• Add supervisor-proc-exit-listener event listener plugin for Supervisor in SwSS Docker container which in turn loads a list of critical processes for which to monitor for unexpected exits.
• Configure swss.service to always auto-restart the service if it stops, with a delay of 30 seconds [Also set a rate limit of 3 restarts within 20 minutes (1200 seconds). If this number of restarts is exceeded, systemd will stop attempting to restart the service and place it in a 'failed' state. To restart the service after entering this state, one must first run systemctl reset-failed [we should probably also call this command in config load_minigraph before restarting services]
• Add systemd dependencies to ensure dependents services also gets restarted along with SwSS
• Also add "WantedBy=swss.service" option to unit files of services which need to be started with SwSS (currently teamd, snmp, dhcp_relay and radv). The "Requires=swss.service" option causes the dependent services to stop and restart along with SwSS (when calling systemctl stop swss.service and systemctl restart swss.service). However this will not cause them to start with SwSS (when calling systemctl start swss.service). This functionality is enabled with the addition of the "WantedBy=" option.
• Also change the way the DHCP relay Docker container waits for interfaces to be ready before starting the relay agent process. Rather than using ip commands, now check STATE_DB for interface entries with "state" == "ok"
• supervisor-proc-exit-listener script resides in files/scripts/ so that the same script can be installed in multiple Docker containers. To add this solution to another container, one simply needs to do the following:
  1. Add the script to the container's "_FILES" variable in the container's Makefile, and ensure it gets copied into the container in the container's Dockerfile
  2. Add a /etc/supervisor/critical_processes file to the container specifying all critical processes, one per line
  3. Add the event listener as a process to the container's supervisor config file

- How it Works

• If a critical process running in the swss container crashes/exits abnormally, supervisor-proc-exit-listener will send a SIGTERM signal to Supervisor, causing it to exit also
• Since Supervisor is running as PID 1 within the Docker container, when Supervisor process exits, it will cause the container to stop
• When the SwSS Docker container stops, systemd will consider the swss.service to have stopped unexpectedly. Systemd will wait 30 seconds, then stop dependent services, restart SwSS, and lastly restart dependent services (currently syncd, teamd, snmp, dhcp_relay, radv and telemetry), unless it has reached the threshold for restarting, in which case it will move the service into a 'failed' state and not attempt to restart it.

- How to verify it

Send a signal to one of the critical processes to cause it to appear to exit abnormally (e.g., pkill -11 orchagent). Ensure the swss, syncd, teamd, snmp, dhcp_relay, radv and telemetry services get restarted per the above details.

NOTE: My updates to systemd dependencies in this PR also fixes #2752

[prsunny]

intfsyncd is no longer present.

[jleveque]

Thanks! I meant to delete that line, but forgot. It won't cause any issues, but I'll open a new PR to remove it soon.

[jleveque]

PR here: #2850