Fix arp/test_unknown_mac by disabling arp_update by justin-wong-ce · Pull Request #13410 · sonic-net/sonic-mgmt

justin-wong-ce · 2024-06-21T22:53:42Z

Description of PR

The DUT will occasionally (~5 mins) send a echo request through IPv6 to the PTF container. When the PTF container replies, it populates the fdb table. This is bad because the test expects the fdb table to be empty.

~~Temporarily disabling IPv6 on the PTF container during the test will eliminate the problem.~~
Temporarily disabling arp_update on the DUT during the test will eliminate the problem.

Summary:
Fixes #

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

Test was flaky. Monitoring tcpdump on the PTF container and the DUT's fdb table shows IPv6 echo is causing mac addresses to be learned on the DUT even though it was previously flushed.

How did you do it?

Disable IPv6.

How did you verify/test it?

Disabling IPv6 will allow the test to consistently past. If IPv6 is re-enabled during the test and packets are sent, it immediately fails.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld · 2024-06-21T22:58:15Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/arp/test_unknown_mac.py:45:1: E302 expected 2 blank lines, found 1 tests/arp/test_unknown_mac.py:47:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:54:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:55:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:56:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:58:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:60:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:61:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:63:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

lolyu

Can we simply ignore the fdb entries from the IPv6 echo replies?

justin-wong-ce · 2024-06-24T23:37:51Z

Can we simply ignore the fdb entries from the IPv6 echo replies?

I don't believe there is a simple way to just ignore it.
The echo reply would cause the DUT to learn all the mac addresses used in the test (even the "fake" ones that are generated).
There is also no telltale sign from show mac that will indicate that a particular entry is from the echo reply.

Even if there is a way to differentiate ipv6 echo replied fdb entries, I don't believe it will be a good idea to filter or add logic to check, as the goal of the test is to test the behaviour when the mac address is unknown. The test will be the most accurate and functionally correct if the DUT indeed does not know the mac at all.

justin-wong-ce · 2024-06-25T22:52:20Z

Can we simply ignore the fdb entries from the IPv6 echo replies?

Actually, will change it so that we disable IPv6 on the DUT side instead of disabling it on the PTF container. Disabling on PTF may cause some previously populated items to be lost.

StormLiangMS · 2024-06-27T04:20:31Z

@lolyu could you help to review?

mssonicbld · 2024-07-04T04:44:22Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/arp/test_unknown_mac.py:44:1: E302 expected 2 blank lines, found 1 tests/arp/test_unknown_mac.py:71:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

justin-wong-ce · 2024-07-04T04:51:50Z

Changed to disable IPv6 from DUT instead of PTF. This method of disabling PTF is same as the method used here:
https://github.com/sonic-net/sonic-mgmt/blob/master/tests/qos/qos_sai_base.py#L1815-L1830

lolyu

Maybe we can disable the arp refresh by disabling the arp_update:

# docker exec -it swss supervisorctl stop arp_update
arp_update: stopped
# docker exec -it swss supervisorctl start arp_update
arp_update: started

justin-wong-ce · 2024-07-12T23:09:06Z

Maybe we can disable the arp refresh by disabling the arp_update:

# docker exec -it swss supervisorctl stop arp_update
arp_update: stopped
# docker exec -it swss supervisorctl start arp_update
arp_update: started

Will use this method instead, thanks.

The DUT will occasionally (~5 mins) send a echo request through IPV6 to the ptf container (from `arp_update`), and this is casuing the fdb table to be populated - which is bad because the test expects the fdb table to be empty. Temporarily disabling arp_update during the test will eliminate the problem.

justin-wong-ce · 2024-07-15T18:12:05Z

Changed the fix method to use the arp_update command instead of disabling ipv6

mssonicbld · 2024-07-15T18:12:28Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/arp/test_unknown_mac.py:44:1: E302 expected 2 blank lines, found 1 tests/arp/test_unknown_mac.py:53:121: E501 line too long (121 > 120 characters) tests/arp/test_unknown_mac.py:57:121: E501 line too long (122 > 120 characters) tests/arp/test_unknown_mac.py:59:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2024-07-15T18:12:31Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/arp/test_unknown_mac.py:44:1: E302 expected 2 blank lines, found 1 tests/arp/test_unknown_mac.py:53:121: E501 line too long (121 > 120 characters) tests/arp/test_unknown_mac.py:57:121: E501 line too long (122 > 120 characters) tests/arp/test_unknown_mac.py:59:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

ZhaohuiS · 2024-07-25T04:59:09Z

@lolyu could you please review it again?

lolyu · 2024-07-26T00:49:17Z

tests/arp/test_unknown_mac.py

+        rand_selected_dut(AnsibleHost) : dut instance
+    """
+    duthost = rand_selected_dut
+    assert duthost.shell("docker exec -t swss supervisorctl stop arp_update")['stdout_lines'][0] \


If the arp_update stop fails here, LINE#59 will not be executed, we'd better to restart arp_update in this case.

That is the intention - fail fast if any other test somehow disabled arp_update, or if it is disabled unintentionally.

Though if you prefer to continue the test, I am open to change it so re-enable it. Please let me know your decision, thanks.

Let's restart it

Changed so it will allow arp_update to remain off and it will restart when the test is done.

lolyu

LGTM

StormLiangMS

LGTM

What is the motivation for this PR? Test was flaky. Monitoring tcpdump on the PTF container and the DUT's fdb table shows IPv6 echo is causing mac addresses to be learned on the DUT even though it was previously flushed. How did you do it? Disable IPv6. How did you verify/test it? Disabling IPv6 will allow the test to consistently past. If IPv6 is re-enabled during the test and packets are sent, it immediately fails. Any platform specific information?

mssonicbld · 2024-08-14T08:45:39Z

Cherry-pick PR to 202405: #14121

What is the motivation for this PR? Test was flaky. Monitoring tcpdump on the PTF container and the DUT's fdb table shows IPv6 echo is causing mac addresses to be learned on the DUT even though it was previously flushed. How did you do it? Disable IPv6. How did you verify/test it? Disabling IPv6 will allow the test to consistently past. If IPv6 is re-enabled during the test and packets are sent, it immediately fails. Any platform specific information?

mssonicbld · 2024-08-22T08:28:27Z

Cherry-pick PR to 202311: #14213

What is the motivation for this PR? Test was flaky. Monitoring tcpdump on the PTF container and the DUT's fdb table shows IPv6 echo is causing mac addresses to be learned on the DUT even though it was previously flushed. How did you do it? Disable IPv6. How did you verify/test it? Disabling IPv6 will allow the test to consistently past. If IPv6 is re-enabled during the test and packets are sent, it immediately fails. Any platform specific information?

lolyu reviewed Jun 23, 2024

View reviewed changes

justin-wong-ce force-pushed the master branch from fee948e to 9c68307 Compare July 4, 2024 04:43

This comment was marked as outdated.

Sign in to view

lolyu reviewed Jul 11, 2024

View reviewed changes

justin-wong-ce closed this Jul 15, 2024

justin-wong-ce force-pushed the master branch from 9c68307 to beeeaa5 Compare July 15, 2024 18:07

justin-wong-ce reopened this Jul 15, 2024

justin-wong-ce changed the title ~~Fix arp/test_unknown_mac by disabling ipv6~~ Fix arp/test_unknown_mac by disabling arp_update Jul 15, 2024

Fix precommit check

388931b

lolyu reviewed Jul 26, 2024

View reviewed changes

Allow disabled arp_update before test

83b479c

lolyu approved these changes Aug 9, 2024

View reviewed changes

StormLiangMS approved these changes Aug 14, 2024

View reviewed changes

StormLiangMS merged commit 390edf0 into sonic-net:master Aug 14, 2024

StormLiangMS added Request for 202311 branch Request for 202405 branch Approved for 202405 branch and removed Request for 202311 branch labels Aug 14, 2024

mssonicbld added the Created PR to 202405 branch label Aug 14, 2024

mssonicbld mentioned this pull request Aug 14, 2024

[action] [PR:13410] Fix arp/test_unknown_mac by disabling arp_update #14121

Merged

8 tasks

mssonicbld added Included in 202405 branch and removed Created PR to 202405 branch labels Aug 14, 2024

StormLiangMS added Request for 202311 branch Approved for 202311 branch labels Aug 22, 2024

mssonicbld added the Created PR to 202311 branch label Aug 22, 2024

mssonicbld mentioned this pull request Aug 22, 2024

[action] [PR:13410] Fix arp/test_unknown_mac by disabling arp_update #14213

Merged

8 tasks

mssonicbld added Included in 202311 branch and removed Created PR to 202311 branch labels Aug 22, 2024

Conversation

justin-wong-ce commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Jun 21, 2024

Uh oh!

lolyu left a comment

Choose a reason for hiding this comment

Uh oh!

justin-wong-ce commented Jun 24, 2024

Uh oh!

justin-wong-ce commented Jun 25, 2024

Uh oh!

StormLiangMS commented Jun 27, 2024

Uh oh!

mssonicbld commented Jul 4, 2024

Uh oh!

justin-wong-ce commented Jul 4, 2024

Uh oh!

This comment was marked as outdated.

lolyu left a comment

Choose a reason for hiding this comment

Uh oh!

justin-wong-ce commented Jul 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justin-wong-ce commented Jul 15, 2024

Uh oh!

mssonicbld commented Jul 15, 2024

Uh oh!

mssonicbld commented Jul 15, 2024

Uh oh!

ZhaohuiS commented Jul 25, 2024

Uh oh!

lolyu Jul 26, 2024

Choose a reason for hiding this comment

Uh oh!

justin-wong-ce Jul 30, 2024

Choose a reason for hiding this comment

Uh oh!

lolyu Aug 8, 2024

Choose a reason for hiding this comment

Uh oh!

justin-wong-ce Aug 8, 2024

Choose a reason for hiding this comment

Uh oh!

lolyu left a comment

Choose a reason for hiding this comment

Uh oh!

StormLiangMS left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Aug 14, 2024

Uh oh!

mssonicbld commented Aug 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

justin-wong-ce commented Jun 21, 2024 •

edited

Loading

justin-wong-ce commented Jul 12, 2024 •

edited

Loading