Adding support for calculating balancing in multi-lc/multi-asic case (Test_fib.py) by vperumal · Pull Request #6391 · sonic-net/sonic-mgmt

vperumal · 2022-09-21T22:06:41Z

Description of PR

Currently in multi-lc/multi-asic environment, expected count is not calculated properly and divided equally over all the interfaces, even though number of paths per asic maybe different. Added support for calculating expected count based on number of interface per asic.

The reason we needed to calculate per asic is because BGP add-path feature is currently not supported. So the current behavior for multi-lc/multi-asic case is traffic gets load balanced per asic from remote end and then load balanced based on the number of interfaces per asic. But since the script currently does a flat calculation based on number of interfaces, script fails for deviation above 25%.

For e.g.

Current behavior - say we have 4 interfaces (2 on asic0, 1 on asic1, 1 on asic2) and send 100 packets. Exp_count is 25 per port. But the traffic is divided 33 packets on asic0, 33 on asic1 and 33 asic2. Further dividing it to 16 packets on asic0 for the two interfaces, causing the testcase to fail.

New behavior - Expected count will be calculated on per_asic level so expected count for asic0 will be 33 and per interface on it to 16.

Summary:
Fixes # (issue)

Type of change

Fill x for your type of change.
Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911
202012

Approach

What is the motivation for this PR?

Currently in multi-lc/multi-asic environment, expected count is not calculated properly and divided equally over all the interfaces, even though number of paths per asic maybe different.

How did you do it?

Added support for calculating expected count based on number of interface per asic.

How did you verify/test it?

Verified it on Cisco-8808 chassis

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

vperumal · 2022-09-21T22:06:57Z

FYI @abdosi

gechiang · 2022-10-03T22:32:49Z

/azp run

azure-pipelines · 2022-10-03T22:33:00Z

Azure Pipelines successfully started running 1 pipeline(s).

vperumal · 2022-10-07T20:13:42Z

@abdosi , As per your request, added the balancing results:

Kindly notice that the expected count is different for the 3 asics (based on interfaces per asic in the dest port list)

14:41:36.115 root 14:41:36.115 root 14:41:36.116 root 14:41:36.116 root 14:41:36.116 root 14:41:36.116 root 14:41:36.116 root 14:41:36.116 root 14:41:36.116 root 14:41:36.116 root 14:41:36.117 root 14:41:36.117 root 14:41:36.117 root 14:41:36.117 root 14:41:36.117 root 14:41:36.118 root 14:41:36.119 root 14:41:36.119 root 14:41:36.119 root 14:41:36.119 root 14:41:36.119 root 14:41:36.119 root 14:41:36.119 root 14:41:36.119 root 14:41:36.120 root 14:41:36.120 root 14:41:36.120 root 14:41:36.120 root 14:41:36.120 root 14:41:36.120 root 14:41:36.120 root 14:41:36.120 root 14:41:36.121 root 14:41:36.121 root 14:41:36.121 root 14:41:36.121 root 14:41:36.121 root 14:41:36.121 root 14:41:36.121 root 14:41:36.121 root 14:41:36.121 root 14:41:36.148 root : INFO : type port(s) exp_cnt act_cnt diff(%)
: INFO : ECMP [0, 1] 1125 1214 7.91%
: INFO : LAG 0 607 605 -0.33%
: INFO : LAG 1 607 609 0.33%
: INFO : ECMP [2, 3] 1125 1118 -0.62%
: INFO : LAG 2 559 542 -3.04%
: INFO : LAG 3 559 576 3.04%
: INFO : ECMP [4, 5] 1125 1120 -0.44%
: INFO : LAG 4 560 541 -3.39%
: INFO : LAG 5 560 579 3.39%
: INFO : ECMP [6, 7] 1125 1066 -5.24%
: INFO : LAG 6 533 554 3.94%
: INFO : LAG 7 533 512 -3.94%
: INFO : ECMP [8, 9] 1125 1119 -0.53%
: INFO : LAG 8 559 558 -0.27%
: INFO : LAG 9 559 561 0.27%
: INFO : ECMP [10, 11] 1125 1113 -1.07%
: INFO : LAG 10 556 548 -1.53%
: INFO : LAG 11 556 565 1.53%
: INFO : ECMP [16, 17] 658 654 -0.7%
: INFO : LAG 16 327 305 -6.73%
: INFO : LAG 17 327 349 6.73%
: INFO : ECMP [13, 14] 658 675 2.49%
: INFO : LAG 13 337 319 -5.48%
: INFO : LAG 14 337 356 5.48%
: INFO : ECMP [12] 658 678 2.95%
: INFO : ECMP [15] 658 650 -1.31%
: INFO : ECMP [18] 658 649 -1.46%
: INFO : ECMP [19] 658 668 1.43%
: INFO : ECMP [20] 658 633 -3.89%
: INFO : ECMP [21] 658 624 -5.25%
: INFO : ECMP [22] 658 693 5.22%
: INFO : ECMP [23] 658 662 0.52%
: INFO : ECMP [24] 833 857 2.88%
: INFO : ECMP [25] 833 883 6.0%
: INFO : ECMP [26] 833 825 -0.96%
: INFO : ECMP [27] 833 822 -1.32%
: INFO : ECMP [28] 833 834 0.12%
: INFO : ECMP [29] 833 799 -4.08%
: INFO : ECMP [30] 833 821 -1.44%
: INFO : ECMP [31] 833 823 -1.2%
: INFO : ** END TEST CASE fib_test.FibTest

abdosi · 2022-10-14T22:51:51Z

@arlakshm for viz.

@vperumal please add more details in PR description what trigger the need of this change.
Also verify Single asic platforms are also fine after this change.

vperumal · 2022-10-17T20:18:58Z

@abdosi, I have updated the PR description as per your request and also verified that the change works on T0-64 and T1-64-lag profiles (Both are tested on single asic platforms)

sanjair-git · 2022-10-24T20:00:08Z

@vperumal, the below import for 'defaultdict' is missing in both files 'fib_test.py', 'hash_test.py' and only if we have this change, the above fix works, and the test cases are going through.

azure-pipelines · 2022-11-08T23:53:08Z

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 ansible/roles/test/files/ptftests/fib_test.py:13:1: E265 block comment should start with '# ' ansible/roles/test/files/ptftests/fib_test.py:15:1: E265 block comment should start with '# ' ansible/roles/test/files/ptftests/fib_test.py:27:1: F401 'ptf.config' imported but unused ansible/roles/test/files/ptftests/fib_test.py:40:1: E302 expected 2 blank lines, found 1 ansible/roles/test/files/ptftests/fib_test.py:68:5: E265 block comment should start with '# ' ansible/roles/test/files/ptftests/fib_test.py:70:5: E265 block comment should start with '# ' ansible/roles/test/files/ptftests/fib_test.py:161:24: F402 import 'fib' from line 37 shadowed by loop variable ansible/roles/test/files/ptftests/fib_test.py:197:121: E501 line too long (138 > 120 characters) ansible/roles/test/files/ptftests/fib_test.py:222:105: E502 the backslash is redundant between brackets ansible/roles/test/files/ptftests/fib_test.py:223:17: E128 continuation line under-indented for visual indent ansible/roles/test/files/ptftests/fib_test.py:237:121: E501 line too long (134 > 120 characters) ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

vperumal · 2022-11-08T23:53:28Z

@vperumal, the below import for 'defaultdict' is missing in both files 'fib_test.py', 'hash_test.py' and only if we have this change, the above fix works, and the test cases are going through.

Thanks @sanjair-git - I have added that change

vperumal · 2022-11-08T23:55:18Z

@abdosi : As per your request, I have made the change applicable only for non-voq based chassis. Kindly take a look and let me know.

arlakshm

As comments

azure-pipelines · 2022-11-09T23:16:30Z

The pre-commit check detected issues in the files touched by this pull request.
The detected issues may be old or new. For new issues, please try to fix them.

For old issues, it is not mandatory to fix them because they were not caused by this change. It is unfair to blame
author of this pull request. But if you can take extra effort to fix the old issues as well, that would be great!

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing ansible/roles/test/files/ptftests/fib_test.py Fixing ansible/roles/test/files/ptftests/py3/hash_test.py fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 ansible/roles/test/files/ptftests/fib_test.py:13:1: E265 block comment should start with '# ' ansible/roles/test/files/ptftests/fib_test.py:15:1: E265 block comment should start with '# ' ansible/roles/test/files/ptftests/fib_test.py:27:1: F401 'ptf.config' imported but unused ansible/roles/test/files/ptftests/fib_test.py:40:1: E302 expected 2 blank lines, found 1 ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

abdosi · 2022-11-22T21:40:06Z

@arlakshm can you please review again

…(Test_fib.py) (#6391) Currently in multi-lc/multi-asic environment, expected count is not calculated properly and divided equally over all the interfaces, even though number of paths per asic maybe different. Added support for calculating expected count based on number of interface per asic. The reason we needed to calculate per asic is because BGP add-path feature is currently not supported. So the current behavior for multi-lc/multi-asic case is traffic gets load balanced per asic from remote end and then load balanced based on the number of interfaces per asic. But since the script currently does a flat calculation based on number of interfaces, script fails for deviation above 25%.

…ic-mgmt into internal-202205 Fix merge conflicts. - Fix verify_no_packet_any call in fib_test (sonic-net#6461) - Fix the test case test_TSA failure when check the routes on the eos host (sonic-net#6483) - Use conditional mark to skip testcase instead of required_mocked_dualtor (sonic-net#6766) - [tagged_arp] fix issue 'fixture ports_list not found' (sonic-net#6773) - [QoS] fixes after moving to python3 (sonic-net#6786) - update parse funciton for image url (sonic-net#6848) - Fix typo in get_queue_counter (sonic-net#6852) - Revert "Fix loganalyzer.py UnicodeDecodeError (sonic-net#6524)" (sonic-net#6858) - Enhancing core_dump_and_config_check to be multi-asic aware (sonic-net#6527) - Adding support for calculating balancing in multi-lc/multi-asic case (Test_fib.py) (sonic-net#6391) - Support different RC in case of pre or post sanity check failed (sonic-net#6860) - Update getbuild.py to support pass an empty access_token - [202205] Fixing auto_techsupport (sonic-net#6882) - Merge branch 'azure-202205' into dev/yaqiangzhu/202205_manually_merge

Adding support for calculating balancing in multi-lc chassis

c74f7c7

vperumal changed the title ~~Adding support for calculating balancing in multi-lc chassis~~ Adding support for calculating balancing in multi-lc/multi-asic case (Test_fib.py) Sep 21, 2022

abdosi previously approved these changes Oct 14, 2022

View reviewed changes

gechiang added the Request for 202205 branch label Oct 14, 2022

vperumal added 2 commits November 8, 2022 15:38

Merge branch 'master' into fib_multi_lc

28b4b94

As requested made the changes for non-voq based chassis

6ea7ff7

vperumal dismissed abdosi’s stale review via 6ea7ff7 November 8, 2022 23:51

arlakshm reviewed Nov 9, 2022

View reviewed changes

Comment thread tests/fib/test_fib.py Outdated

Comment thread ansible/roles/test/files/ptftests/py3/hash_test.py Outdated

Comment thread ansible/roles/test/files/ptftests/py3/hash_test.py

Comment thread ansible/roles/test/files/ptftests/fib_test.py Outdated

arlakshm requested a review from wangxin November 9, 2022 01:22

Updated - based on comments received during chassis meeting

75ad3d7

abdosi approved these changes Nov 22, 2022

View reviewed changes

arlakshm approved these changes Nov 23, 2022

View reviewed changes

abdosi merged commit 10c19cb into sonic-net:master Nov 23, 2022

wangxin added the Included in 202205 branch label Nov 24, 2022

Conversation

vperumal commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

vperumal commented Sep 21, 2022

Uh oh!

gechiang commented Oct 3, 2022

Uh oh!

azure-pipelines bot commented Oct 3, 2022

Uh oh!

vperumal commented Oct 7, 2022

Uh oh!

abdosi commented Oct 14, 2022

Uh oh!

vperumal commented Oct 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sanjair-git commented Oct 24, 2022

Uh oh!

azure-pipelines bot commented Nov 8, 2022

Uh oh!

vperumal commented Nov 8, 2022

Uh oh!

vperumal commented Nov 8, 2022

Uh oh!

arlakshm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

azure-pipelines bot commented Nov 9, 2022

Uh oh!

abdosi commented Nov 22, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vperumal commented Sep 21, 2022 •

edited

Loading

vperumal commented Oct 17, 2022 •

edited

Loading