Skip to content

[Services] Restart Telemetry service upon unexpected critical process exit. #3768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jleveque merged 7 commits into from
Nov 19, 2019
Merged

[Services] Restart Telemetry service upon unexpected critical process exit. #3768

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2214842
[docker-telemetry] Create a file named critical_processes.
yozhao101 Nov 14, 2019
3f87a21
[docker-telemetry] Add paths of supervisord listener script and critical
yozhao101 Nov 14, 2019
144fada
[docer-telemetry] Make event listener autostart by adding option in
yozhao101 Nov 14, 2019
178b24d
[docker-telemetry] Configure systemd to stop restarting telemetry if it
yozhao101 Nov 14, 2019
0830a3f
[docker-telemetry] Add macro $(SUPERVISOR_PROC_EXIT_LISTENER_SCRIPT)
yozhao101 Nov 14, 2019
0d42d43
[docker-telemetry] Modify the supervisord conf file to enable the
yozhao101 Nov 15, 2019
cee94b7
[docker-telemetry] Modify the auto-restart option from false to true in
yozhao101 Nov 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions dockers/docker-sonic-telemetry/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,5 +35,7 @@ RUN apt-get clean -y && \

COPY ["start.sh", "telemetry.sh", "dialout.sh", "/usr/bin/"]
COPY ["supervisord.conf", "/etc/supervisor/conf.d/"]
COPY ["files/supervisor-proc-exit-listener", "/usr/bin"]
COPY ["critical_processes", "/etc/supervisor"]

ENTRYPOINT ["/usr/bin/supervisord"]
2 changes: 2 additions & 0 deletions dockers/docker-sonic-telemetry/critical_processes
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
telemetry
dialout
12 changes: 9 additions & 3 deletions dockers/docker-sonic-telemetry/supervisord.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@ logfile_maxbytes=1MB
logfile_backups=2
nodaemon=true

[eventlistener:supervisor-proc-exit-listener]
command=/usr/bin/supervisor-proc-exit-listener
events=PROCESS_STATE_EXITED
autostart=always
autorestart=unexpected

[program:start.sh]
command=/usr/bin/start.sh
priority=1
Expand All @@ -14,7 +20,7 @@ stderr_logfile=syslog
[program:rsyslogd]
command=/usr/sbin/rsyslogd -n
priority=2
autostart=false
autostart=true
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
Expand All @@ -23,14 +29,14 @@ stderr_logfile=syslog
command=/usr/bin/telemetry.sh
priority=3
autostart=false
autorestart=true
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog

[program:dialout]
command=/usr/bin/dialout.sh
priority=4
autostart=false
autorestart=true
autorestart=false
stdout_logfile=syslog
stderr_logfile=syslog
4 changes: 4 additions & 0 deletions files/build_templates/telemetry.service.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,16 @@ Description=Telemetry container
Requires=database.service
After=database.service
Before=ntp-config.service
StartLimitIntervalSec=1200
StartLimitBurst=3

[Service]
User={{ sonicadmin_user }}
ExecStartPre=/usr/bin/{{docker_container_name}}.sh start
ExecStart=/usr/bin/{{docker_container_name}}.sh wait
ExecStop=/usr/bin/{{docker_container_name}}.sh stop
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target
1 change: 1 addition & 0 deletions rules/telemetry.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
SONIC_TELEMETRY = sonic-telemetry_0.1_$(CONFIGURED_ARCH).deb
$(SONIC_TELEMETRY)_SRC_PATH = $(SRC_PATH)/telemetry
SONIC_DPKG_DEBS += $(SONIC_TELEMETRY)
$(SONIC_TELEMETRY)_FILES += $(SUPERVISOR_PROC_EXIT_LISTENER_SCRIPT)