Skip to content

Commit c651a9a

Browse files
authored
[dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 (#7083)
To prevent error [messages](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802) like the following from being logged: ``` Mar 17 02:33:48.523153 vlab-01 INFO swss#supervisord 2021-03-17 02:33:48,518 ERRO pool supervisor-proc-exit-listener event buffer overflowed, discarding event 46 ``` This is basically an addendum to #5247, which increased the event buffer size for dependent-startup. While supervisor-proc-exit-listener doesn't subscribe to as many events as dependent-startup, there is still a chance some containers (like swss, as in the example above) have enough processes running to cause an overflow of the default buffer size of 10. This is especially important for preventing erroneous log_analyzer failures in the sonic-mgmt repo regression tests, which have started occasionally causing PR check builds to fail. Example [here](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802). I set all supervisor-proc-exit-listener event buffer sizes to 1024, and also updated all dependent-startup event buffer sizes to 1024, as well, to keep things simple, unified, and allow headroom so that we will not need to adjust these values frequently, if at all.
1 parent bcff251 commit c651a9a

File tree

33 files changed

+60
-27
lines changed

33 files changed

+60
-27
lines changed

dockers/docker-database/supervisord.conf.j2

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ command=/usr/bin/supervisor-proc-exit-listener --container-name database
88
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
99
autostart=true
1010
autorestart=unexpected
11+
buffer_size=1024
1112

1213
[program:rsyslogd]
1314
command=/usr/sbin/rsyslogd -n -iNONE

dockers/docker-dhcp-relay/docker-dhcp-relay.supervisord.conf.j2

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,14 @@ autorestart=unexpected
1010
startretries=0
1111
exitcodes=0,3
1212
events=PROCESS_STATE
13-
buffer_size=50
13+
buffer_size=1024
1414

1515
[eventlistener:supervisor-proc-exit-listener]
1616
command=/usr/bin/supervisor-proc-exit-listener --container-name dhcp_relay
1717
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
1818
autostart=true
1919
autorestart=unexpected
20+
buffer_size=1024
2021

2122
[program:rsyslogd]
2223
command=/usr/sbin/rsyslogd -n -iNONE

dockers/docker-fpm-frr/frr/supervisord/supervisord.conf.j2

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,14 @@ autorestart=unexpected
1010
startretries=0
1111
exitcodes=0,3
1212
events=PROCESS_STATE
13-
buffer_size=50
13+
buffer_size=1024
1414

1515
[eventlistener:supervisor-proc-exit-listener]
1616
command=/usr/bin/supervisor-proc-exit-listener --container-name bgp
1717
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
1818
autostart=true
1919
autorestart=unexpected
20+
buffer_size=1024
2021

2122
[program:rsyslogd]
2223
command=/usr/sbin/rsyslogd -n -iNONE

dockers/docker-fpm-gobgp/supervisord.conf

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ command=/usr/bin/supervisor-proc-exit-listener --container-name bgp
88
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
99
autostart=true
1010
autorestart=unexpected
11+
buffer_size=1024
1112

1213
[program:start.sh]
1314
command=/usr/bin/start.sh

dockers/docker-fpm-quagga/supervisord.conf

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ command=/usr/bin/supervisor-proc-exit-listener --container-name bgp
88
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
99
autostart=true
1010
autorestart=unexpected
11+
buffer_size=1024
1112

1213
[program:start.sh]
1314
command=/usr/bin/start.sh

dockers/docker-iccpd/supervisord.conf

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ autorestart=unexpected
1010
startretries=0
1111
exitcodes=0,3
1212
events=PROCESS_STATE
13+
buffer_size=1024
1314

1415
[program:rsyslogd]
1516
command=/usr/sbin/rsyslogd -n -iNONE

dockers/docker-lldp/supervisord.conf.j2

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,14 @@ autorestart=unexpected
1010
startretries=0
1111
exitcodes=0,3
1212
events=PROCESS_STATE
13-
buffer_size=25
13+
buffer_size=1024
1414

1515
[eventlistener:supervisor-proc-exit-listener]
1616
command=/usr/bin/supervisor-proc-exit-listener --container-name lldp
1717
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
1818
autostart=true
1919
autorestart=unexpected
20+
buffer_size=1024
2021

2122
[program:rsyslogd]
2223
command=/usr/sbin/rsyslogd -n -iNONE

dockers/docker-macsec/supervisord.conf

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,14 @@ autorestart=unexpected
1010
startretries=0
1111
exitcodes=0,3
1212
events=PROCESS_STATE
13-
buffer_size=25
13+
buffer_size=1024
1414

1515
[eventlistener:supervisor-proc-exit-listener]
1616
command=/usr/bin/supervisor-proc-exit-listener --container-name macsec
1717
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
1818
autostart=true
1919
autorestart=unexpected
20+
buffer_size=1024
2021

2122
[program:rsyslogd]
2223
command=/usr/sbin/rsyslogd -n -iNONE
@@ -35,4 +36,4 @@ autorestart=false
3536
stdout_logfile=syslog
3637
stderr_logfile=syslog
3738
dependent_startup=true
38-
dependent_startup_wait_for=rsyslogd:running
39+
dependent_startup_wait_for=rsyslogd:running

dockers/docker-nat/supervisord.conf

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,14 @@ autorestart=unexpected
1010
startretries=0
1111
exitcodes=0,3
1212
events=PROCESS_STATE
13-
buffer_size=25
13+
buffer_size=1024
1414

1515
[eventlistener:supervisor-proc-exit-listener]
1616
command=/usr/bin/supervisor-proc-exit-listener --container-name nat
1717
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
1818
autostart=true
1919
autorestart=unexpected
20+
buffer_size=1024
2021

2122
[program:rsyslogd]
2223
command=/usr/sbin/rsyslogd -n -iNONE

dockers/docker-orchagent/supervisord.conf

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,14 @@ autorestart=unexpected
1010
startretries=0
1111
exitcodes=0,3
1212
events=PROCESS_STATE
13-
buffer_size=100
13+
buffer_size=1024
1414

1515
[eventlistener:supervisor-proc-exit-listener]
1616
command=/usr/bin/supervisor-proc-exit-listener --container-name swss
1717
events=PROCESS_STATE_EXITED,PROCESS_STATE_RUNNING
1818
autostart=true
1919
autorestart=unexpected
20+
buffer_size=1024
2021

2122
[program:rsyslogd]
2223
command=/usr/sbin/rsyslogd -n -iNONE

0 commit comments

Comments
 (0)