[rsyslog]: Use RELP instead of UDP for forwarding from container to host #18113
+121
−163
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,24 +9,20 @@ | |
| #### MODULES #### | ||
| ################# | ||
|
|
||
| # | ||
| # Set a rate limit on messages from the container | ||
| # | ||
| module(load="imuxsock" SysSock.RateLimit.Interval="300" SysSock.RateLimit.Burst="20000") # provides support for local system logging | ||
| #module(load="imklog") # provides kernel logging support | ||
| #module(load="immark") # provides --MARK-- message capability | ||
|
|
||
| # provides UDP syslog reception | ||
| #module(load="imudp") | ||
| #input(type="imudp" port="514") | ||
|
|
||
| # provides TCP syslog reception | ||
| #module(load="imtcp") | ||
| #input(type="imtcp" port="514") | ||
|
|
||
|
|
||
| ########################### | ||
|
|
@@ -37,7 +33,8 @@ set $.CONTAINER_NAME=getenv("CONTAINER_NAME"); | |
|
|
||
| # Set remote syslog server | ||
| template (name="ForwardFormatInContainer" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %$.CONTAINER_NAME%#%syslogtag%%msg:::sp-if-no-1st-sp%%msg%") | ||
| module(load="omrelp") | ||
| *.* action(type="omrelp" target=`echo $SYSLOG_TARGET_IP` port="2514" action.resumeRetryCount="60" queue.type="LinkedList" queue.size="20000" Template="ForwardFormatInContainer") | ||
|
There was a problem hiding this comment. In this bookworm base rsyslog.conf, the new RELP forwarding action uses action.resumeRetryCount="60" while other container configs in this PR (bullseye/trixie/pmon and rsyslog-container.conf.j2) use "-1" (infinite). With RELP + a LinkedList queue, limiting retries to 60 can still drop container logs if the host rsyslog is restarted or unreachable for longer than that, which undercuts the PR’s reliability goal. Consider aligning this to "-1" (or clearly justify/standardize a finite retry policy across all base images). |
||
|
|
||
| # | ||
| # Use traditional timestamp format. | ||
|
|
@@ -75,4 +72,4 @@ $RepeatedMsgReduction on | |
|
|
||
| ############### | ||
| #### RULES #### | ||
| ############### | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -13,40 +13,34 @@ | |||||||||||||||||
| #### MODULES #### | ||||||||||||||||||
| ################# | ||||||||||||||||||
|
|
||||||||||||||||||
| {% set gconf = (SYSLOG_CONFIG | d({})).get('GLOBAL', {}) -%} | ||||||||||||||||||
| {% set rate_limit_interval = gconf.get('rate_limit_interval') -%} | ||||||||||||||||||
| {% set rate_limit_burst = gconf.get('rate_limit_burst') -%} | ||||||||||||||||||
|
|
||||||||||||||||||
| module(load="imuxsock" {% if rate_limit_interval is not none %}SysSock.RateLimit.Interval="{{ rate_limit_interval }}"{% endif %} {% if rate_limit_burst is not none %}SysSock.RateLimit.Burst="{{ rate_limit_burst }}"{% endif %}) # provides support for local system logging | ||||||||||||||||||
| module(load="imklog" ParseKernelTimestamp="off" KeepKernelTimestamp="on") # provides kernel logging support | ||||||||||||||||||
| #module(load="immark") # provides --MARK-- message capability | ||||||||||||||||||
| {% if syslog_counter and syslog_counter == "true" %} | ||||||||||||||||||
| module(load="omprog") | ||||||||||||||||||
| {% endif %} | ||||||||||||||||||
|
|
||||||||||||||||||
| # provides UDP syslog reception | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. @saiarcot895 This UDP syslog is for remote server?
Contributor
Author
There was a problem hiding this comment. Yes, in the case of a remote syslog server sending over UDP. |
||||||||||||||||||
| module(load="imudp") | ||||||||||||||||||
| input(type="imudp" address="{{udp_server_ip}}" port="514") | ||||||||||||||||||
| {% if docker0_ip and docker0_ip != "" %} | ||||||||||||||||||
| input(type="imudp" address="{{docker0_ip}}" port="514") | ||||||||||||||||||
| {% endif%} | ||||||||||||||||||
|
Comment on lines
27
to
32
There was a problem hiding this comment. The rendered rsyslog.conf sample outputs used by unit tests are currently inconsistent with this template’s active UDP inputs (module(load="imudp") + input(... address=...)). In the repo’s sample_output files, the UDP section is commented out and also lacks the address parameter, so test_j2files.py::test_rsyslog_conf* is likely to fail. Either update the template to comment out UDP reception (if no longer needed) or regenerate/update the sample_output/py3 rsyslog*.conf files so they match the template output. |
||||||||||||||||||
|
|
||||||||||||||||||
| # provides TCP syslog reception | ||||||||||||||||||
| #module(load="imtcp") | ||||||||||||||||||
| #input(type="imtcp" port="514") | ||||||||||||||||||
|
|
||||||||||||||||||
| # provides RELP syslog reception | ||||||||||||||||||
| module(load="imrelp") | ||||||||||||||||||
| input(type="imrelp" address="{{udp_server_ip}}" port="2514") | ||||||||||||||||||
| {% if docker0_ip and docker0_ip != "" %} | ||||||||||||||||||
| input(type="imrelp" address="{{docker0_ip}}" port="2514") | ||||||||||||||||||
| {% endif%} | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
||||||||||||||||||
| ########################### | ||||||||||||||||||
|
|
@@ -61,7 +55,7 @@ $UDPServerRun 514 | |||||||||||||||||
| #$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat | ||||||||||||||||||
|
|
||||||||||||||||||
| # Define a custom template | ||||||||||||||||||
| $template SONiCFileFormat,"%TIMESTAMP:::date-year% %TIMESTAMP%.%TIMESTAMP:::date-subseconds% {{ hostname }} %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" | ||||||||||||||||||
| $ActionFileDefaultTemplate SONiCFileFormat | ||||||||||||||||||
| $template SONiCForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% {{ hostname }} %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" | ||||||||||||||||||
| $template SONiCForwardFormatWithOsVersion,"<%PRI%>%TIMESTAMP:::date-rfc3339% {{ hostname }} {{os_version}} %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" | ||||||||||||||||||
|
Comment on lines
+58
to
61
There was a problem hiding this comment. This template changes SONiCFileFormat to use %TIMESTAMP% (message-reported time) instead of %timegenerated% (local receipt time), but the checked-in expected outputs under src/sonic-config-engine/tests/sample_output/py3 still use %timegenerated%. Please regenerate/update the expected rsyslog.conf and rsyslog_with_docker0.conf outputs accordingly; otherwise the rsyslog.conf Jinja unit tests will fail.
Suggested change
|
||||||||||||||||||
|
|
@@ -125,7 +119,7 @@ template(name="WelfRemoteFormat" type="string" string="%TIMESTAMP% id=firewall t | |||||||||||||||||
| :msg, {{ fmodifier }}ereregex, "{{ regex }}" | ||||||||||||||||||
| {% endif %} | ||||||||||||||||||
| *.{{ severity }} | ||||||||||||||||||
| action(type="omfwd" Target="{{ server }}" Port="{{ port }}" Protocol="{{ proto }}" Template="{{ template }}" action.resumeRetryCount="60" queue.type="LinkedList" queue.size="20000"{{ options }}) | ||||||||||||||||||
| {% endfor %} | ||||||||||||||||||
|
|
||||||||||||||||||
| {% if syslog_counter and syslog_counter == "true" %} | ||||||||||||||||||
|
|
||||||||||||||||||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@saiarcot895 can you mention this syntax change in PR description
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.