Skip to content
Open
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
e348e26
[rsyslog]: Use RELP instead of UDP for forwarding from container to host
saiarcot895 Feb 5, 2024
e275fd2
Try to fix repeated messages being logged, and clean up templates
saiarcot895 Feb 6, 2024
af610f2
Fix errors about $SystemLogRateLimitInterval and $SystemLogRateLimitB…
saiarcot895 Feb 17, 2024
1df7538
Modify omrelp to store messages in memory if the server is unavailable
saiarcot895 Feb 17, 2024
7c937cc
Merge remote-tracking branch 'origin/master' into rsyslog-use-relp
saiarcot895 Jun 7, 2024
dd7af0a
Update rsyslog.conf file for Bookworm
saiarcot895 Jun 9, 2024
6ebca5f
Merge remote-tracking branch 'origin/master' into rsyslog-use-relp
saiarcot895 Jun 9, 2024
ed01b28
Install rsyslog-relp for Bookworm
saiarcot895 Jun 10, 2024
bbe163b
Use new syntax for specifying rate limiting interval and burst
saiarcot895 Jun 13, 2024
e030511
Revert template change to match what is currently on master branch
saiarcot895 Jun 13, 2024
ab995b0
Restore the RepeatedMsgReduction config option
saiarcot895 Jun 21, 2024
d57c5ba
Merge remote-tracking branch 'origin/master' into rsyslog-use-relp
saiarcot895 Jul 27, 2024
4d8056b
Update rsyslog.conf file for pmon docker
saiarcot895 Jul 30, 2024
bf9cda9
Remove extra newlines
saiarcot895 Jul 30, 2024
d5edef8
Add (or re-add) retry support into the default `/etc/rsyslog.conf` files
saiarcot895 Jul 30, 2024
20e8621
If tests fail, print the diff as the error message
saiarcot895 Sep 18, 2024
5310d94
Merge remote-tracking branch 'origin/master' into rsyslog-use-relp
saiarcot895 Oct 2, 2024
5444c45
Merge remote-tracking branch 'origin/master' into rsyslog-use-relp
saiarcot895 Oct 22, 2024
dce24d6
Set a limit of 20000 messages to be queued
saiarcot895 Oct 23, 2024
d115668
Merge remote-tracking branch 'origin/master' into rsyslog-use-relp
saiarcot895 Feb 23, 2026
ed6fc69
Reenable UDP reception and enable retry for forwarding out syslogs
saiarcot895 Feb 25, 2026
b539ebb
Update test output files
saiarcot895 Mar 25, 2026
7213eda
Further update test output files
saiarcot895 Mar 25, 2026
051ea49
Handle case of the UDP server IP and Docker IP being the same
saiarcot895 Mar 30, 2026
b5dcf18
Fix file name
saiarcot895 Mar 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions build_debian.sh
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,7 @@ sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y in
squashfs-tools \
$bootloader_packages \
rsyslog \
rsyslog-relp \
screen \
hping3 \
tcptraceroute \
Expand Down
1 change: 1 addition & 0 deletions dockers/docker-base-bookworm/Dockerfile.j2
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ RUN apt update && \
python-is-python3 \
vim-tiny \
rsyslog \
rsyslog-relp \
# Install redis-tools
redis-tools \
# common dependencies
Expand Down
23 changes: 10 additions & 13 deletions dockers/docker-base-bookworm/etc/rsyslog.conf
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,20 @@
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging

#
# Set a rate limit on messages from the container
#
$SystemLogRateLimitInterval 300
$SystemLogRateLimitBurst 20000

#$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
module(load="imuxsock" SysSock.RateLimit.Interval="300" SysSock.RateLimit.Burst="20000") # provides support for local system logging
#module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability
Comment on lines +15 to +17
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saiarcot895 can you mention this syntax change in PR description

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#module(load="imtcp")
#input(type="imtcp" port="514")


###########################
Expand All @@ -37,7 +33,8 @@ set $.CONTAINER_NAME=getenv("CONTAINER_NAME");

# Set remote syslog server
template (name="ForwardFormatInContainer" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %$.CONTAINER_NAME%#%syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
*.* action(type="omfwd" target=`echo $SYSLOG_TARGET_IP` port="514" protocol="udp" Template="ForwardFormatInContainer")
module(load="omrelp")
*.* action(type="omrelp" target=`echo $SYSLOG_TARGET_IP` port="2514" action.resumeRetryCount="-1" queue.type="LinkedList" Template="ForwardFormatInContainer")
Copy link
Copy Markdown
Contributor

@prgeor prgeor Aug 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saiarcot895 2514 is the port used by relp?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no standard port used by RELP. The default port that rsyslog uses is 514, but that can conflict with regular syslog forwarding over TCP. A couple of the examples in the documentation for this feature uses 2514.


#
# Use traditional timestamp format.
Expand Down Expand Up @@ -75,4 +72,4 @@ $RepeatedMsgReduction on

###############
#### RULES ####
###############
###############
2 changes: 1 addition & 1 deletion dockers/docker-base-bullseye/Dockerfile.j2
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ RUN apt-get update && \

# default rsyslog version is 8.2110.0 which has a bug on log rate limit,
# use backport version 8.2206.0-1~bpo11+1
RUN apt-get -t bullseye-backports -y install rsyslog
RUN apt-get -t bullseye-backports -y install rsyslog rsyslog-relp

# Upgrade pip via PyPI and uninstall the Debian version
RUN pip3 install --upgrade pip
Expand Down
23 changes: 10 additions & 13 deletions dockers/docker-base-bullseye/etc/rsyslog.conf
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saiarcot895 are we still using bullseye?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, bullseye is still being used for a couple containers.

Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,20 @@
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging

#
# Set a rate limit on messages from the container
#
$SystemLogRateLimitInterval 300
$SystemLogRateLimitBurst 20000

#$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
module(load="imuxsock" SysSock.RateLimit.Interval="300" SysSock.RateLimit.Burst="20000") # provides support for local system logging
#module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#module(load="imtcp")
#input(type="imtcp" port="514")


###########################
Expand All @@ -37,7 +33,8 @@ set $.CONTAINER_NAME=getenv("CONTAINER_NAME");

# Set remote syslog server
template (name="ForwardFormatInContainer" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %$.CONTAINER_NAME%#%syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
*.* action(type="omfwd" target=`echo $SYSLOG_TARGET_IP` port="514" protocol="udp" Template="ForwardFormatInContainer")
module(load="omrelp")
*.* action(type="omrelp" target=`echo $SYSLOG_TARGET_IP` port="2514" action.resumeRetryCount="-1" queue.type="LinkedList" Template="ForwardFormatInContainer")

#
# Use traditional timestamp format.
Expand Down Expand Up @@ -75,4 +72,4 @@ $RepeatedMsgReduction on

###############
#### RULES ####
###############
###############
23 changes: 10 additions & 13 deletions dockers/docker-platform-monitor/etc/rsyslog.conf
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,20 @@
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging

#
# Set a rate limit on messages from the container
#
$SystemLogRateLimitInterval 300
$SystemLogRateLimitBurst 20000

#$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
module(load="imuxsock" SysSock.RateLimit.Interval="300" SysSock.RateLimit.Burst="20000") # provides support for local system logging
#module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#module(load="imtcp")
#input(type="imtcp" port="514")


###########################
Expand All @@ -47,7 +43,8 @@ if $programname contains "sensord" and $msg contains "Error getting sensor data:

# Set remote syslog server
template (name="ForwardFormatInContainer" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %$.CONTAINER_NAME%#%syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
*.* action(type="omfwd" target=`echo $SYSLOG_TARGET_IP` port="514" protocol="udp" Template="ForwardFormatInContainer")
module(load="omrelp")
*.* action(type="omrelp" target=`echo $SYSLOG_TARGET_IP` port="2514" action.resumeRetryCount="-1" queue.type="LinkedList" Template="ForwardFormatInContainer")

#
# Use traditional timestamp format.
Expand Down Expand Up @@ -85,4 +82,4 @@ $RepeatedMsgReduction on

###############
#### RULES ####
###############
###############
41 changes: 10 additions & 31 deletions files/image_config/rsyslog/rsyslog-container.conf.j2
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging

#
# Set a rate limit on messages from the container
#
Expand All @@ -26,27 +24,17 @@ $ModLoad imuxsock # provides support for local system logging
{% endif %}
{% endif %}

{% if rate_limit_interval is defined %}
$SystemLogRateLimitInterval {{ rate_limit_interval }}
{% else %}
$SystemLogRateLimitInterval 300
{% endif %}
{% if rate_limit_burst is defined %}
$SystemLogRateLimitBurst {{ rate_limit_burst }}
{% else %}
$SystemLogRateLimitBurst 20000
{% endif %}

#$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
module(load="imuxsock" SysSock.RateLimit.Interval="{{ rate_limit_interval|default('300') }}" SysSock.RateLimit.Burst="{{ rate_limit_burst|default('20000') }}") # provides support for local system logging
#module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#module(load="imtcp")
#input(type="imtcp" port="514")


###########################
Expand All @@ -71,17 +59,8 @@ if ($.PLATFORM == "x86_64-mlnx_msn2700-r0" or $.PLATFORM == "x86_64-mlnx_msn2700

# Set remote syslog server
template (name="ForwardFormatInContainer" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %$.CONTAINER_NAME%#%syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
*.* action(type="omfwd" target=`echo $SYSLOG_TARGET_IP` port="514" protocol="udp" Template="ForwardFormatInContainer")

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Define a custom template
$template SONiCFileFormat,"%TIMESTAMP%.%timestamp:::date-subseconds% %HOSTNAME% %syslogseverity-text:::uppercase% %$.CONTAINER_NAME%#%syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate SONiCFileFormat
module(load="omrelp")
*.* action(type="omrelp" target=`echo $SYSLOG_TARGET_IP` port="2514" action.resumeRetryCount="-1" queue.type="LinkedList" Template="ForwardFormatInContainer")

#
# Set the default permissions for all log files.
Expand Down Expand Up @@ -109,4 +88,4 @@ $RepeatedMsgReduction on

###############
#### RULES ####
###############
###############
38 changes: 16 additions & 22 deletions files/image_config/rsyslog/rsyslog.conf.j2
Original file line number Diff line number Diff line change
Expand Up @@ -13,34 +13,28 @@
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging

{% set gconf = (SYSLOG_CONFIG | d({})).get('GLOBAL', {}) -%}
{% set rate_limit_interval = gconf.get('rate_limit_interval') %}
{% set rate_limit_burst = gconf.get('rate_limit_burst') %}
{% set rate_limit_interval = gconf.get('rate_limit_interval') -%}
{% set rate_limit_burst = gconf.get('rate_limit_burst') -%}

{% if rate_limit_interval is not none %}
$SystemLogRateLimitInterval {{ rate_limit_interval }}
{% endif %}
{% if rate_limit_burst is not none %}
$SystemLogRateLimitBurst {{ rate_limit_burst }}
{% endif %}

$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
module(load="imuxsock" {% if rate_limit_interval is not none %}SysSock.RateLimit.Interval="{{ rate_limit_interval }}"{% endif %} {% if rate_limit_burst is not none %}SysSock.RateLimit.Burst="{{ rate_limit_burst }}"{% endif %}) # provides support for local system logging
module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability

# provides UDP syslog reception
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saiarcot895 This UDP syslog is for remote server?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, in the case of a remote syslog server sending over UDP.

$ModLoad imudp
$UDPServerAddress {{udp_server_ip}} #bind to localhost before udp server run
$UDPServerRun 514
{% if docker0_ip and docker0_ip != "" %}
$UDPServerAddress {{docker0_ip}}
$UDPServerRun 514
{% endif%}
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#module(load="imtcp")
#input(type="imtcp" port="514")

# provides RELP syslog reception
module(load="imrelp")
input(type="imrelp" address="{{udp_server_ip}}" port="2514")
{% if docker0_ip and docker0_ip != "" %}
input(type="imrelp" address="{{docker0_ip}}" port="2514")
{% endif%}


###########################
Expand Down
22 changes: 11 additions & 11 deletions src/sonic-config-engine/tests/sample_output/py3/rsyslog.conf
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,21 @@
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging



$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
module(load="imuxsock" ) # provides support for local system logging
module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability

# provides UDP syslog reception
$ModLoad imudp
$UDPServerAddress 1.1.1.1 #bind to localhost before udp server run
$UDPServerRun 514
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#module(load="imtcp")
#input(type="imtcp" port="514")

# provides RELP syslog reception
module(load="imrelp")
input(type="imrelp" address="1.1.1.1" port="2514")

Comment on lines 20 to 31
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expected output still has UDP reception commented out (and without address=...), but files/image_config/rsyslog/rsyslog.conf.j2 currently renders active imudp inputs with an address and port. Update/regenerate this sample output to match the template output (and also adjust the SONiCFileFormat timestamp line in this file to reflect the template change to %TIMESTAMP%).

Copilot uses AI. Check for mistakes.

###########################
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,22 @@
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging



$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
module(load="imuxsock" ) # provides support for local system logging
module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability

# provides UDP syslog reception
$ModLoad imudp
$UDPServerAddress 1.1.1.1 #bind to localhost before udp server run
$UDPServerRun 514
$UDPServerAddress 2.2.2.2
$UDPServerRun 514
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#module(load="imtcp")
#input(type="imtcp" port="514")

# provides RELP syslog reception
module(load="imrelp")
input(type="imrelp" address="1.1.1.1" port="2514")
input(type="imrelp" address="2.2.2.2" port="2514")

Comment on lines 20 to 33
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as rsyslog.conf: this expected output keeps the UDP reception commented out and missing address=, but rsyslog.conf.j2 renders active imudp inputs (including both udp_server_ip and docker0_ip). Please regenerate/update this sample output to match the template output (and also update the SONiCFileFormat timestamp line in this file to use %TIMESTAMP% per the template).

Copilot uses AI. Check for mistakes.

###########################
Expand Down
8 changes: 4 additions & 4 deletions src/sonic-config-engine/tests/test_j2files.py
Original file line number Diff line number Diff line change
Expand Up @@ -865,8 +865,8 @@ def test_rsyslog_conf(self):
pattern = r'^action.*Device="eth0".*'
for line in file:
assert not bool(re.match(pattern, line.strip())), "eth0 is not allowed in Mgfx device"
self.assertTrue(utils.cmp(os.path.join(self.test_dir, 'sample_output', utils.PYvX_DIR, 'rsyslog.conf'),
self.output_file))
expected = os.path.join(self.test_dir, 'sample_output', utils.PYvX_DIR, 'rsyslog.conf')
self.assertTrue(utils.cmp(expected, self.output_file), self.run_diff(expected, self.output_file))

def test_rsyslog_conf_docker0_ip(self):
if utils.PYvX_DIR != 'py3':
Expand All @@ -881,8 +881,8 @@ def test_rsyslog_conf_docker0_ip(self):

argument = ['-j', config_db_json, '-t', conf_template, '-a', additional_data]
self.run_script(argument, output_file=self.output_file)
self.assertTrue(utils.cmp(os.path.join(self.test_dir, 'sample_output', utils.PYvX_DIR,
'rsyslog_with_docker0.conf'), self.output_file))
expected = os.path.join(self.test_dir, 'sample_output', utils.PYvX_DIR, 'rsyslog_with_docker0.conf')
Copy link

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_rsyslog_conf_same_ip passes docker0_ip equal to udp_server_ip, and rsyslog.conf.j2 explicitly avoids adding a second input(...) when the IPs are the same. The expected fixture should therefore be the single-IP output (looks like rsyslog_same_ip.conf was added for this), not rsyslog_with_docker0.conf which contains duplicate inputs. As written, this test will fail or won’t validate the intended behavior.

Suggested change
expected = os.path.join(self.test_dir, 'sample_output', utils.PYvX_DIR, 'rsyslog_with_docker0.conf')
expected = os.path.join(self.test_dir, 'sample_output', utils.PYvX_DIR, 'rsyslog_same_ip.conf')

Copilot uses AI. Check for mistakes.
self.assertTrue(utils.cmp(expected, self.output_file), self.run_diff(expected, self.output_file))

def tearDown(self):
os.environ["CFGGEN_UNIT_TESTING"] = ""
Expand Down