Skip to content

[ZMQ] Ignore loganalyzer during ZMQ test#17018

Merged
qiluo-msft merged 5 commits intosonic-net:masterfrom
liuh-80:dev/liuh/ignore_zmq_test_log
Feb 24, 2025
Merged

[ZMQ] Ignore loganalyzer during ZMQ test#17018
qiluo-msft merged 5 commits intosonic-net:masterfrom
liuh-80:dev/liuh/ignore_zmq_test_log

Conversation

@liuh-80
Copy link
Contributor

@liuh-80 liuh-80 commented Feb 19, 2025

Disable LogAnalyzer during ZMQ test

Why I did it

The test case zmq/test_gnmi_zmq.py failed because LogAnalyzer found some error message, which is not related with ZMQ feature.

The test case is GNMI test, not touch any syncd related feature:

On 720dt has following error, which already confirmed fixed on latest release:
expected_missing_match: 0 Match Messages: 2025 Jan 20 17:50:16.879657 bjw2-can-720dt-1 ERR syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_query:509 Invalid sai_api_t 536870912 passed to sai_api_query 2025 Jan 20 17:50:16.879657 bjw2-can-720dt-1 ERR syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_query:509 Invalid sai_api_t 536870913 passed to sai_api_query 2025 Jan 20 17:50:16.879657 bjw2-can-720dt-1 ERR syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_query:509 Invalid sai_api_t 536870926 passed to sai_api_query 2025 Jan 20 17:50:16.885941 bjw2-can-720dt-1 ERR syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_query:509 Invalid sai_api_t 536870914 passed to sai_api_query 2025 Jan 20 17:50:16.889221 bjw2-can-720dt-1 ERR syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_query:509 Invalid sai_api_t 536870915 passed to sai_api_query 2025 Jan 20 17:

On mellanox 2700, which confirmed a known issue:
E 2025 Feb 19 06:50:56.694627 strtk5-msn2700-02 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2073]- get_dispatch_attribs_handler: Failed Get #0, EGRESS_SAMPLE_MIRROR_SESSION, key:PORT [OID:0x1001D00000001] [log_port:0x1001D]
E
E 2025 Feb 19 06:50:56.843363 strtk5-msn2700-02 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2073]- get_dispatch_attribs_handler: Failed Get #0, INGRESS_SAMPLE_MIRROR_SESSION, key:PORT [OID:0x1001F00000001] [log_port:0x1001F]
E
E 2025 Feb 19 06:50:56.843363 strtk5-msn2700-02 ERR syncd#SDK: [SAI_UTILS.ERR] ./src/mlnx_sai_utils.c[2073]- get_dispatch_attribs_handler: Failed Get #0, EGRESS_SAMPLE_MIRROR_SESSION, key:PORT [OID:0x1001F00000001] [log_port:0x1001F]

Work item tracking
  • Microsoft ADO: 30980895

How I did it

Disable LogAnalyzer during ZMQ test

How to verify it

Pass all test case.

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106
  • 202111
  • 202205
  • 202211
  • 202305

Tested branch (Please provide the tested image version)

Description for the changelog

Disable LogAnalyzer during ZMQ test

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

@mssonicbld
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@liuh-80 liuh-80 marked this pull request as ready for review February 20, 2025 00:02
@liuh-80 liuh-80 requested a review from qiluo-msft February 20, 2025 00:02
@liuh-80 liuh-80 changed the title Ignore loganalyzer during ZMQ test [ZMQ] Ignore loganalyzer during ZMQ test Feb 20, 2025
@qiluo-msft qiluo-msft requested a review from Blueve February 21, 2025 01:44
@yaqiangz
Copy link
Contributor

Arista has addressed the issue, such error log is not seen in Arista 720DT now



pytestmark = [
pytest.mark.disable_loganalyzer,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arista has fixed this error log in newest release.

Do we have clue on why this test could trigger errors on other platforms?
We usually prefer to fix the error log or suppress the specific error log pattern in log analyzer instead of disabling it.
But it would make sense to disable it if the test itself could trigger error logs by nature (such as restart some critical services).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test will reload config during test, which will restart all service. I think this is the reason why the error been triggered.
There are 2 kinds of error break this test, so if we don't disable log analyzer there may similar issue break this test again.



pytestmark = [
pytest.mark.disable_loganalyzer,
Copy link
Contributor

@yaqiangz yaqiangz Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a suggestion, if error log is generated by config reload for known reason, then we could disable loganalyzer when reloading:

    loganalyzer[duthost.hostname].add_start_ignore_mark()
    config_reload()
    loganalyzer[duthost.hostname].add_end_ignore_mark()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will not work, because many service not fully start after config_reload command finish, it will take some time, on some slow planform for example Mellanox 2700, it will take few minutes.

@qiluo-msft qiluo-msft merged commit ee118ee into sonic-net:master Feb 24, 2025
12 checks passed
nnelluri-cisco pushed a commit to nnelluri-cisco/sonic-mgmt that referenced this pull request Mar 15, 2025
Disable LogAnalyzer during ZMQ test

Why I did it
The test case zmq/test_gnmi_zmq.py failed because LogAnalyzer found some error message, which is not related with ZMQ feature.

The test case is GNMI test, not touch any syncd related feature
@lizhijianrd
Copy link
Contributor

I've verified this backport on Nokia-7215 Mx 202411 branch.
This PR is essential for zmq test pass on Nokia-7215.

mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Mar 20, 2025
Disable LogAnalyzer during ZMQ test

Why I did it
The test case zmq/test_gnmi_zmq.py failed because LogAnalyzer found some error message, which is not related with ZMQ feature.

The test case is GNMI test, not touch any syncd related feature
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202411: #17623

mssonicbld pushed a commit that referenced this pull request Mar 22, 2025
Disable LogAnalyzer during ZMQ test

Why I did it
The test case zmq/test_gnmi_zmq.py failed because LogAnalyzer found some error message, which is not related with ZMQ feature.

The test case is GNMI test, not touch any syncd related feature
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants