Skip to content

Fix issue with test_syslog_rate_limit#10867

Merged
StormLiangMS merged 1 commit intosonic-net:masterfrom
Staphylo:master-fix-syslog-rate-limit
Dec 8, 2023
Merged

Fix issue with test_syslog_rate_limit#10867
StormLiangMS merged 1 commit intosonic-net:masterfrom
Staphylo:master-fix-syslog-rate-limit

Conversation

@Staphylo
Copy link
Copy Markdown
Contributor

Description of PR

There is a race condition in the rate-limit-host test code.
Changing the rate limit via the cli triggers hostcfgd to restart rsyslogd. However there is no wait to make sure this actually happens. This leads to some flakyness where loganalyzer.py would try to put a marker while the rsyslog systemd service is being restarted and crash.

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Back port request

  • 201911
  • 202012
  • 202205
  • 202305

Approach

What is the motivation for this PR?

Fix flaky test to enhance pass rate reliability.

How did you do it?

Reproduced on my testbed, investigated the issue, fixed the issue

How did you verify/test it?

Stressed the test overnight without further failure

@mssonicbld
Copy link
Copy Markdown
Collaborator

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/syslog/test_syslog_rate_limit.py:180:121: E501 line too long (124 > 120 characters)
tests/syslog/test_syslog_rate_limit.py:249:1: E302 expected 2 blank lines, found 1
tests/syslog/test_syslog_rate_limit.py:253:1: E302 expected 2 blank lines, found 1
tests/syslog/test_syslog_rate_limit.py:273:1: E302 expected 2 blank lines, found 1

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

There is a race condition in the rate-limit-host
Changing the rate limit via the cli triggers hostcfgd to restart rsyslogd.
However there is no wait to make sure this actually happens.
This leads to some flakyness where loganalyzer.py would try to put a marker while
the rsyslog systemd service is being restarted and crash.
@Staphylo Staphylo force-pushed the master-fix-syslog-rate-limit branch from fd117f3 to f0e7f71 Compare November 28, 2023 09:18
@Staphylo
Copy link
Copy Markdown
Contributor Author

@StormLiangMS

Copy link
Copy Markdown
Collaborator

@StormLiangMS StormLiangMS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@StormLiangMS StormLiangMS merged commit e93a983 into sonic-net:master Dec 8, 2023
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Dec 8, 2023
There is a race condition in the rate-limit-host
Changing the rate limit via the cli triggers hostcfgd to restart rsyslogd.
However there is no wait to make sure this actually happens.
This leads to some flakyness where loganalyzer.py would try to put a marker while
the rsyslog systemd service is being restarted and crash.
@mssonicbld
Copy link
Copy Markdown
Collaborator

Cherry-pick PR to 202305: #10982

mssonicbld pushed a commit that referenced this pull request Dec 8, 2023
There is a race condition in the rate-limit-host
Changing the rate limit via the cli triggers hostcfgd to restart rsyslogd.
However there is no wait to make sure this actually happens.
This leads to some flakyness where loganalyzer.py would try to put a marker while
the rsyslog systemd service is being restarted and crash.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants