Skip to content

[autorestart] address log analyzer issues in autorestart test #2615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yxieca merged 2 commits into from
Dec 3, 2020
Merged

[autorestart] address log analyzer issues in autorestart test #2615

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
729a53d
[autorestart] address log analyzer issues in autorestart test
yxieca Dec 2, 2020
df62fba
ignore unimplemented API errors
yxieca Dec 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions ansible/roles/test/files/tools/loganalyzer/loganalyzer_common_ignore.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ r, ".* ERR ntpd.*routing socket reports: No buffer space available.*"
r, ".* INFO ntpd.*kernel reports TIME_ERROR: 0x41: Clock Unsynchronized.*"
r, ".* ERR liblogging-stdlog: omfwd: error 11 sending via udp: Resource temporarily unavailable.*"
r, ".* ERR syncd#syncd: brcm_sai_get_port_stats:.* port stats get failed with error.*"
r, ".* ERR pmon#xcvrd.*for checking mux_cable .* side, eeprom read returned a size.*"
r, ".* ERR pmon#ledd.*initializeGlobalConfig: Sonic database config global file.*"
r, ".* NOTICE kernel:.*profile=""/usr/sbin/ntpd"" name=""sbin"" pid=.* comm=""ntpd"" requested_mask=.*"
r, ".* ERR snmp#snmp-subagent.*"
r, ".* ERR route_check.py.*"
Expand Down
66 changes: 30 additions & 36 deletions tests/autorestart/test_container_autorestart.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@


@pytest.fixture(autouse=True)
def ignore_expected_loganalyzer_exception(loganalyzer):
def ignore_expected_loganalyzer_exception(loganalyzer, enum_dut_feature):
"""
Ignore expected failure/error messages during testing the autorestart feature.

Expand Down Expand Up @@ -55,42 +55,36 @@ def ignore_expected_loganalyzer_exception(loganalyzer):
SNMP/TEAMD container hits the limitation of restart. route_check.py also wrote an error message into syslog.

"""
monit_ignoreRegex = [
".*ERR monit.*",
]
swss_ignoreRegex = [
".*ERR swss#orchagent.*removeLag.*",
]
pmon_ignoreRegex = [
".*ERR pmon#xcvrd.*initializeGlobalConfig.*",
".*ERR pmon#thermalctld.*Caught exception while initializing thermal manager.*",
]
syncd_ignoreRegex = [
".*ERR syncd#syncd.*driverEgressMemoryUpdate.*",
".*ERR syncd#syncd.*brcm_sai*",
".*WARNING syncd#syncd.*saiDiscover: skipping since it causes crash.*",
]
teamd_ignoreRegex = [
".*ERR swss#portsyncd.*readData.*netlink reports an error=-33 on reading a netlink socket.*",
]
systemd_ignoreRegex = [
".*ERR systemd.*Failed to start .* container*",
]
kernel_ignoreRegex = [
".*ERR kernel.*PortChannel.*",
]
other_ignoreRegex = [
".*ERR route_check.*",
]
swss_syncd_teamd_regex = [
".*ERR swss#orchagent.*removeLag.*",
".*ERR syncd#syncd.*driverEgressMemoryUpdate.*",
".*ERR syncd#syncd.*brcm_sai*",
".*ERR syncd#syncd.*SAI_API_UNSPECIFIED:sai_api_query.*",
".*WARNING syncd#syncd.*skipping since it causes crash.*",
".*ERR swss#portsyncd.*readData.*netlink reports an error=-33 on reading a netlink socket.*",
]
ignore_regex_dict = {
'common' : [
".*ERR monit.*",
".*ERR systemd.*Failed to start .* container*",
".*ERR kernel.*PortChannel.*",
".*ERR route_check.*",
],
'pmon' : [
".*ERR pmon#xcvrd.*initializeGlobalConfig.*",
".*ERR pmon#thermalctld.*Caught exception while initializing thermal manager.*",
],
'swss' : swss_syncd_teamd_regex,
'syncd' : swss_syncd_teamd_regex,
'teamd' : swss_syncd_teamd_regex,
}

_, feature = decode_dut_port_name(enum_dut_feature)

if loganalyzer:
loganalyzer.ignore_regex.extend(monit_ignoreRegex)
loganalyzer.ignore_regex.extend(swss_ignoreRegex)
loganalyzer.ignore_regex.extend(pmon_ignoreRegex)
loganalyzer.ignore_regex.extend(syncd_ignoreRegex)
loganalyzer.ignore_regex.extend(teamd_ignoreRegex)
loganalyzer.ignore_regex.extend(systemd_ignoreRegex)
loganalyzer.ignore_regex.extend(kernel_ignoreRegex)
loganalyzer.ignore_regex.extend(other_ignoreRegex)
loganalyzer.ignore_regex.extend(ignore_regex_dict['common'])

@judyjoseph judyjoseph Dec 3, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A minor comment, will the common set of match regex be added multiple times when ignore_expected_loganalyzer_exception() is called per enum_dut_feature ?

@judyjoseph judyjoseph Dec 3, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignore my comment Ying .. I think it should be ok as it is parameterized and will be invoked now separately per enum_dut_feature.

@yxieca yxieca Dec 3, 2020

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are fine. the loganalyzer is setup per testcase.

if feature in ignore_regex_dict:
loganalyzer.ignore_regex.extend(ignore_regex_dict[feature])


def get_group_program_info(duthost, container_name, group_name):
Expand Down