[loganalyzer]add log patterns to the common ignore by StormLiangMS · Pull Request #5411 · sonic-net/sonic-mgmt

StormLiangMS · 2022-03-28T03:07:04Z

Description of PR

There are PR tests failed due to loganalyzer, some of the error logs should be ignored by loganalyzer, in order to unblock the PR procedure, move those errors to common ignore for now, and filed an issue to have code owner to check later. #5410

Summary:
Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911
202012

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

wangxin · 2022-03-28T07:27:10Z

There are more error messages need to be temporarily ignored to unblock PR testing:

LogAnalyzerError: {'match_messages': {'/tmp/syslog.vlab-01.2022-03-28-04:46:34': ['Mar 28 04:46:24.861033 vlab-01 ERR sonic_yang: exceptionList:[]\n', "Mar 28 04:46:24.862357 vlab-01 ERR sonic_yang: Data Loading Failed:All Keys are not parsed in TACPLUS#012dict_keys(['global'])\n"]}, 'total': {'expected_match': 0, 'expected_missing_match': 0, 'match': 2}, 'match_files': {'/tmp/syslog.vlab-01.2022-03-28-04:46:34': {'expected_match': 0, 'match': 2}}, 'expect_messages': {'/tmp/syslog.vlab-01.2022-03-28-04:46:34': []}, 'unused_expected_regexp': []}

wen587 · 2022-03-28T09:34:16Z

There are more error messages need to be temporarily ignored to unblock PR testing:

LogAnalyzerError: {'match_messages': {'/tmp/syslog.vlab-01.2022-03-28-04:46:34': ['Mar 28 04:46:24.861033 vlab-01 ERR sonic_yang: exceptionList:[]\n', "Mar 28 04:46:24.862357 vlab-01 ERR sonic_yang: Data Loading Failed:All Keys are not parsed in TACPLUS#012dict_keys(['global'])\n"]}, 'total': {'expected_match': 0, 'expected_missing_match': 0, 'match': 2}, 'match_files': {'/tmp/syslog.vlab-01.2022-03-28-04:46:34': {'expected_match': 0, 'match': 2}}, 'expect_messages': {'/tmp/syslog.vlab-01.2022-03-28-04:46:34': []}, 'unused_expected_regexp': []}

GCU test's log analyzer failure will be fixed by this PR.
#5391

wen587 · 2022-03-29T14:29:30Z

Hi @wangxin @StormLiangMS , Seems the added regex are all GCU regex. Currently they are all covered in GCU ignored regex.
You can ignore the errors in (Test kvmtest-t0-part2) as the test have passed after the PR merge #5411

wangxin · 2022-03-30T01:03:32Z

@wen587 Do you mean PR #5391? Let's try to get this PR passed to unblock PR testing asap.

wen587 · 2022-03-30T01:35:44Z

Hi @wangxin , yes I mean PR #5391. It has been merged.
So you can rebase the recent change. The t0-part2 error should be gone.

… files (#5193)" (#5433) This reverts commit 03cccf7. Reverts #5193 After this fix was merged, PR test keeps failing because of errors in syslog. We spent some effort trying to temporarily ignore the errors. However, the list seems endless. Please refer to: [loganalyzer]add log patterns to the common ignore #5411 Add loganalyzer ignore regex for GCU #5391 We need a way to temporarily unblock PR testing. Let's revert this fix for now. Then I'll submit another PR to fix the loganalyzer issue together with a complete ignore list.

wangxin · 2022-03-30T06:44:05Z

The log analyzer fix was reverted. Let's abandon this one. I'll submit a new PR to include both the log analyzer fix and the changes of this PR.

Loganalyzer was broken in PR sonic-net#3235. The issue is that common config was loaded in subprocess for adding marks to syslog. After the subprocess exited, the common config is lost. PR sonic-net#5193 tried to fix this issue. However, because of many new error logs sneaked in when log analyzer was not working, PR testing started to fail by these error logs after PR sonic-net#5193 was merged. PR sonic-net#5191 and sonic-net#5411 tried to workaround the PR testing failure to unblock PR testing. PR sonic-net#5191 is to address the GCU related error logs and was merged. PR sonic-net#5411 tried to add other error logs to the common ignore list. But the effort took too long because the ignore list seemed endless. To unblock PR testing as soon as possible, the original fix sonic-net#5193 was reverted in sonic-net#5433. This PR tries to complete the work left over from sonic-net#5411 and sonic-net#5433. Changes: 1. Fix the log analyzer common config not loaded issue. 2. Temporarily add error logs to the common ignore list. 3. Improve the logging of log analyzer and parallel_run 4. PR testing t0_part2 takes much more time than t0_part1 after the GCU test scripts are added. This change re-balanced t0 part1&part2 testing by moving some of the tests from part2 to part1. 5. Sorted the PR testing scripts in alphabetic order. Signed-off-by: Xin Wang <xiwang5@microsoft.com>

What is the motivation for this PR? Loganalyzer was broken in PR #3235. The issue is that common config was loaded in subprocess for adding marks to syslog. After the subprocess exited, the common config is lost. PR #5193 tried to fix this issue. However, because of many new error logs sneaked in when log analyzer was not working, PR testing started to fail by these error logs after PR #5193 was merged. PR #5391 and #5411 tried to work around the PR testing failure to unblock PR testing. PR #5391 is to address the GCU related error logs and was merged. PR #5411 tried to add other error logs to the common ignore list. But the effort took too long because the ignore list seemed endless. To unblock PR testing as soon as possible, the original fix #5193 was reverted in #5433. This PR tries to complete the work left over from #5411 and #5433. How did you do it? Changes: * Fix the log analyzer common config not loaded issue. * Temporarily add error logs to the common ignore list. * Improve the logging of log analyzer and parallel_run * PR testing t0_part2 takes much more time than t0_part1 after the GCU test scripts are added. This change re-balanced t0 part1&part2 testing by moving some of the tests from part2 to part1. * Sorted the PR testing scripts in alphabetic order. How did you verify/test it? Tested run a few test scripts with log analyzer enabled on KVM testbed. Signed-off-by: Xin Wang <xiwang5@microsoft.com>

… files (sonic-net#5193)" (sonic-net#5433) This reverts commit 03cccf7. Reverts sonic-net#5193 After this fix was merged, PR test keeps failing because of errors in syslog. We spent some effort trying to temporarily ignore the errors. However, the list seems endless. Please refer to: [loganalyzer]add log patterns to the common ignore sonic-net#5411 Add loganalyzer ignore regex for GCU sonic-net#5391 We need a way to temporarily unblock PR testing. Let's revert this fix for now. Then I'll submit another PR to fix the loganalyzer issue together with a complete ignore list.

add log patterns to the common ignore

3b2783c

StormLiangMS requested a review from a team as a code owner March 28, 2022 03:07

StormLiangMS changed the title ~~add log patterns to the common ignore~~ [loganalyzer]add log patterns to the common ignore Mar 28, 2022

StormLiangMS and others added 4 commits March 28, 2022 10:34

skip fail to find all failed cases

d946f58

enahnce ignore for gcu

89c7610

ignore some error

d082be7

Ignore the error of failed to start dhcp_relay container

8a6f117

sumukhatv added the Request for 202012 branch label Mar 29, 2022

Ignore the rsyslog error syslog

c67aee7

wangxin mentioned this pull request Mar 30, 2022

Revert "Fix LogAnalyzer to force load regular expressions from common files" #5433

Merged

wangxin mentioned this pull request Mar 30, 2022

Fix loganalyzer no common config issue and re-balance PR testing #5435

Merged

5 tasks

StormLiangMS closed this Apr 1, 2022

wangxin removed the Request for 202012 branch label Apr 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[loganalyzer]add log patterns to the common ignore#5411

[loganalyzer]add log patterns to the common ignore#5411
StormLiangMS wants to merge 6 commits intosonic-net:masterfrom
StormLiangMS:log_ignore

StormLiangMS commented Mar 28, 2022 •

edited

Loading

Uh oh!

wangxin commented Mar 28, 2022

Uh oh!

wen587 commented Mar 28, 2022

Uh oh!

wen587 commented Mar 29, 2022

Uh oh!

wangxin commented Mar 30, 2022

Uh oh!

wen587 commented Mar 30, 2022 •

edited

Loading

Uh oh!

wangxin commented Mar 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

StormLiangMS commented Mar 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

wangxin commented Mar 28, 2022

Uh oh!

wen587 commented Mar 28, 2022

Uh oh!

wen587 commented Mar 29, 2022

Uh oh!

wangxin commented Mar 30, 2022

Uh oh!

wen587 commented Mar 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangxin commented Mar 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

StormLiangMS commented Mar 28, 2022 •

edited

Loading

wen587 commented Mar 30, 2022 •

edited

Loading