[Mellanox] SKU creator Tool by madhanmellanox · Pull Request #1163 · sonic-net/sonic-utilities

madhanmellanox · 2020-10-12T01:42:37Z

What I did
I added a new script file sonic_sku_create.py to generate a new HWSKU based on XML file or Minigraph. But, focus is on creating HWSKU based on XML file and not on the minigraph and l2 mode. I also added a unit test pyTest script sku_create_test.py in sonic-utilities-test folder to test the script in Unit Testing mode.

Motivation: To create SKU for Mellanox platforms based on XML file with Port related inputs and also through Minigraph file.
This tool also allows to split a port or unsplit ports based on configuration which modifies the port related information in port_config.ini and config_db.json

usage: sonic_sku_create.py [-h] [-v]
(-f FILE | -m [MINIGRAPH_FILE] | -j JSON_FILE | -pp PORT_SPLIT PORT_SPLIT)
[-b BASE] [-r] [-k HWSKU] [-p] [-vv]

Create a new SKU

optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
-f FILE, --file FILE SKU definition from xml file. -f OR -m or -j must be provided when creating a new SKU
-m [MINIGRAPH_FILE], --minigraph_file [MINIGRAPH_FILE]
SKU definition from minigraph file. -f OR -m or -j must be provided when creating a new SKU
-j JSON_FILE, --json_file JSON_FILE
SKU definition from config_db.json file. -f OR -m OR -j must be provided when creating a new SKU
-pp PORT_SPLIT PORT_SPLIT, --port_split PORT_SPLIT PORT_SPLIT
port name and split
-b BASE, --base BASE SKU base definition
-r, --remove Remove SKU folder
-k HWSKU, --hwsku HWSKU
SKU name to be used when creating a new SKU or for L2 configuration mode
-p, --print Print port_config.ini without creating a new SKU
-vv, --verbose Verbose output

Output:
Creation of Port_config.ini file.
Modifying Port_config.ini and config_db.json based on port split and unsplit configuration.

How I did it
I did it by adding these files which are required for new SKU generation in the forthcoming releases.
How to verify it
I verified all the options of sonic_sku_create.py file and tested them thoroughly as well as created a test case in sku_create_test.py file to test the functionality of sonic_sku_create.py in Unit Test mode.

**- Note we only backport fixes to a release branch, not features!

Please also provide a reason for the backporting below.**
201911 - This feature will be used in 201911..
-->

Previous command output (if the output of a command-line utility has changed)
New command output (if the output of a command-line utility has changed)

admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py -f Mellanox-SN2700-C28D10.xml
MSN2700 -  even front panel ports ( 32 ) are not allowed to split by 4
Error - Illegal split by 4
admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py -f Mellanox-SN2700-C28D11.xml
SKU directory: /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-C28D11/ already exists
 Please use -r flag to remove the SKU dir first
admin@r-qa-sw-eth-2134:/tmp$
admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py -r -f Mellanox-SN2700-C28D11.xml
You are about to permanently delete the SKU /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-C28D11/ !!
Do you want to continue (Yes/No)?Yes
SKU directory: /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-C28D11/ was removed
admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py  -f Mellanox-SN2700-C28D11.xml
Created a new sku (Location: /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-C28D11/)
admin@r-qa-sw-eth-2134:/tmp$
admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py -m ams20-0101-0623-08t0.xml
SKU directory: /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-D48C8/ already exists
 Please use -r flag to remove the SKU dir first
admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py -r -m ams20-0101-0623-08t0.xml
You are about to permanently delete the SKU /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-D48C8/ !!
Do you want to continue (Yes/No)?Yes
SKU directory: /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-D48C8/ was removed
admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py -m ams20-0101-0623-08t0.xml
Created a new sku (Location: /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-D48C8/)
admin@r-qa-sw-eth-2134:/tmp$
admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py -j config_db_fra.json
Created a new sku (Location: /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-test/)
admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py -pp Ethernet0 4x25
['name', 'lanes', 'alias', 'index']
port_config.ini file does not contain all fields, Exiting...
admin@r-qa-sw-eth-2134:/tmp$
admin@r-qa-sw-eth-2134:/tmp$ sudo python sonic_sku_create.py -pp Ethernet0 4x25 -vv
ARGS:  Namespace(base=None, file=None, hwsku=None, json_file=None, minigraph_file=None, port_split=['Ethernet0', '4x25'], print=False, remove=False, verbose=True)
['name', 'lanes', 'alias', 'index', 'speed']
msn2700_specific -> Removing  ['Ethernet5', [1, 2, 3, 4], 'etp2a', '5', '50000']
msn2700_specific -> Removing  ['Ethernet6', [1, 2, 3, 4], 'etp2b', '6', '50000']
MSN2700 - Front panel port  2  should be removed due to port  1 Split by 4
Error - Illegal split by 4
admin@r-qa-sw-eth-2134:/tmp$
madhan@arc-build-server:/builds2/madhan/utilitiescloneskucreate0714/sonic-utilities/sonic-utilities-tests$ sudo pytest -v sku_create_test.py -s
==================================================================== test session starts ====================================================================
platform linux2 -- Python 2.7.6, pytest-3.0.0, py-1.5.2, pluggy-0.3.1 -- /usr/bin/python2
cachedir: .cache
rootdir: /builds2/madhan/utilitiescloneskucreate0714/sonic-utilities/sonic-utilities-tests, inifile: pytest.ini
collected 1 items

sku_create_test.py::TestSkuCreate::test_no_param Created a new sku (Location: /builds2/madhan/utilitiescloneskucreate0714/sonic-utilities/sonic-utilities-tests/sku_create_input/Mellanox-SN2700-D48C8_NEW/)

('Output file: ', '/builds2/madhan/utilitiescloneskucreate0714/sonic-utilities/sonic-utilities-tests/sku_create_input/Mellanox-SN2700-D48C8_NEW/port_config.ini', 'exists. SUCCESS!')
('Output file: ', '/builds2/madhan/utilitiescloneskucreate0714/sonic-utilities/sonic-utilities-tests/sku_create_input/Mellanox-SN2700-D48C8_NEW/port_config.ini', ' and model file: ', '/builds2/madhan/utilitiescloneskucreate0714/sonic-utilities/sonic-utilities-tests/sku_create_input/Mellanox-SN2700-D48C8/port_config.ini', 'contents are same. SUCCESS!')
PASSEDTEARDOWN


================================================================= 1 passed in 0.47 seconds ==================================================================
madhan@arc-build-server:/builds2/madhan/utilitiescloneskucreate0714/sonic-utilities/sonic-utilities-tests$

lgtm-com · 2020-10-12T01:48:19Z

This pull request introduces 14 alerts when merging 5c64dc43c2dfee31207959c11b6c7f5b791ee9e2 into 561d133 - view on LGTM.com

new alerts:

7 for Unused local variable
4 for Testing equality to None
1 for Unused import
1 for Redundant assignment
1 for Result of integer division may be truncated

…reate a new HWSKU based on XML file. Also added file sku_create_test.py pyTest unit test script to test the same

lgtm-com · 2020-10-13T16:03:19Z

This pull request introduces 1 alert when merging 58e9708 into 3a7457c - view on LGTM.com

new alerts:

1 for First parameter of a method is not named 'self'

madhanmellanox · 2020-10-13T17:49:26Z

@lguohan can someone from MSFT review this SKU creator tool PR?

madhanmellanox · 2020-10-13T20:48:42Z

retest

madhanmellanox · 2020-10-14T01:35:15Z

retest this please

madhanmellanox · 2020-10-14T04:13:20Z

retest this please

madhanmellanox · 2020-10-20T17:08:28Z

@qiluo-msft This is the PR for SKU creator tool. Can you please review this?

qiluo-msft · 2020-10-20T20:04:30Z

scripts/sonic_sku_create.py

+
+optional arguments:
+  -h, --help            Show this help message and exit
+  -v, --version         show program's version number and exit


show [](start = 24, length = 4)

Capitalize #Closed

addressed in next commit

qiluo-msft · 2020-10-20T20:05:21Z

scripts/sonic_sku_create.py

@@ -0,0 +1,805 @@
+#! /usr/bin/env python


python [](start = 16, length = 6)

Please make sure the same script could be run by python3 #Closed

tabulate module used in the script does not work with python3. It says module not found.

So if this is true, please explicitly use python2 in shebang line.

In reply to: 508805018 [](ancestors = 508805018)

Addressed in next commit

qiluo-msft · 2020-10-20T20:06:37Z

scripts/sonic_sku_create.py

+minigraph_ns1 = "http://schemas.datacontract.org/2004/07/Microsoft.Search.Autopilot.Evolution"
+INTERFACE_KEY = "Ethernet"
+
+test_envir = None


Could you remove test related code out of this script? #Closed

Yes, I will remove test related code of the script. So, I may have to remove the pytest for testing the SKU creator tool also. Is that okay?

qiluo-msft · 2020-10-20T20:08:40Z

scripts/sonic_sku_create.py

+### port_config.ini header
+PORTCONFIG_HEADER = ["# name", "lanes", "alias", "index", "speed"]
+platform_4 = ['x86_64-mlnx_lssn2700-r0','x86_64-mlnx_msn2010-r0','x86_64-mlnx_msn2100-r0','x86_64-mlnx_msn2410-r0','x86_64-mlnx_msn2700-r0','x86_64-mlnx_msn2740-r0','x86_64-mlnx_msn3700c-r0','x86_64-mlnx_msn3700-r0','x86_64-mlnx_msn3800-r0']
+platform_8 = ['x86_64-mlnx_msn4600c-r0','x86_64-mlnx_msn4700-r0']


Why we have mlnx platforms? I thought this script is general to any platform. #Closed

This script as of now is for creating SKU for Mellanox Platforms. To make SKUs for other platforms, we may have to bring in the details of other platforms and process specific to that platform, because, port name, lane number and other restrictions may change according to respective platform.

Parsing of xml file or minigraph file for SKU definition does not have any Mellanox related definitions.

qiluo-msft · 2020-10-20T20:15:38Z

scripts/sonic_sku_create.py

+                    idx +=1
+
+        if match is None :
+            raise ValueError("Couldn't find a SKU ",hwsku, "in minigraph file")


, [](start = 51, length = 1)

Add one blank char after , #Closed

qiluo-msft · 2020-10-20T20:16:11Z

scripts/sonic_sku_create.py

+                self.parse_deviceinfo(child,hwsku)
+
+    def check_json_lanes_with_bko(self, data, port_idx):
+        #Function to find matching entry in bko_dict that matches Port details from config_db.json file


[](start = 8, length = 1)

Add one blank char after # #Closed

Addressed in next commit

qiluo-msft · 2020-10-20T20:17:02Z

scripts/sonic_sku_create.py

+                if "speed" in curr_port_dict:
+                    curr_speed = curr_port_dict.get("speed")
+                else:
+                    print(curr_port_str, "does not contain speed key, Exiting...")


print [](start = 20, length = 5)

print error messages to stderr #Closed

qiluo-msft · 2020-11-04T19:32:42Z

scripts/sonic_sku_create.py

+    group.add_argument('-f', '--file', action='store', nargs=1, help='SKU definition from xml file. -f OR -m or -j must be provided when creating a new SKU', default=None)
+    group.add_argument('-m', '--minigraph_file', action='store', nargs='?', help='SKU definition from minigraph file. -f OR -m or -j must be provided when creating a new SKU', const="/etc/sonic/minigraph.xml")
+    group.add_argument('-j', '--json_file', action='store', nargs=1, help='SKU definition from config_db.json file. -f OR -m OR -j must be provided when creating a new SKU', default=None)
+    group.add_argument('-pp', '--port_split', action='store', nargs=2, help='port name and split', default=None)


-pp [](start = 24, length = 3)

Linux command convention is using single char in short option name. In this case, it's ok to have only long option name.
And one more below. #Closed

Addressed in next commit.

qiluo-msft · 2020-11-04T19:33:10Z

sonic-utilities-tests/sku_create_test.py

+        return True
+
+    def test_no_param(self):
+        my_command = sku_create_script + " -f "  + sku_def_file  + " -d " + input_path + " -pl " + "x86_64-mlnx_msn2700-r0"


-pl [](start = 91, length = 3)

Could you also test without -pl ? Which should be more common use case. #Closed

Addressed in next commit.

lgtm-com · 2020-11-04T20:46:30Z

This pull request introduces 1 alert when merging 9eb7121 into acfa824 - view on LGTM.com

new alerts:

1 for Unused local variable

jleveque · 2020-11-06T00:43:16Z

@madhanmellanox: We renamed the sonic-utilities-tests directory to tests. Can you please move these tests into the new directory and remove sonic-utilities-tests?

@liat-grozovik, @qiluo-msft for visibility.

madhanmellanox · 2020-11-12T16:05:43Z

@jleveque can this SKU creator tool PR be cherry picked to 201911? Or I have to create a new PR for 201911?

jleveque · 2020-11-12T18:00:20Z

@madhanmellanox: This PR might cherry-pick cleanly, since the directory is still called sonic-utilities-tests in 201911. However, this PR doesn't have your test fix to delete the new SKU directory before running the tests. Maybe you should open a separate PR which combines the two?

I added a new script file sonic_sku_create.py to generate a new HWSKU based on XML file or Minigraph. But, focus is on creating HWSKU based on XML file and not on the minigraph and l2 mode. I also added a unit test pyTest script sku_create_test.py in sonic-utilities-test folder to test the script in Unit Testing mode. Motivation: To create SKU for Mellanox platforms based on XML file with Port related inputs and also through Minigraph file. This tool also allows to split a port or unsplit ports based on configuration which modifies the port related information in port_config.ini and config_db.json usage: sonic_sku_create.py [-h] [-v] (-f FILE | -m [MINIGRAPH_FILE] | -j JSON_FILE | -pp PORT_SPLIT PORT_SPLIT) [-b BASE] [-r] [-k HWSKU] [-p] [-vv] Create a new SKU optional arguments: -h, --help show this help message and exit -v, --version show program's version number and exit -f FILE, --file FILE SKU definition from xml file. -f OR -m or -j must be provided when creating a new SKU -m [MINIGRAPH_FILE], --minigraph_file [MINIGRAPH_FILE] SKU definition from minigraph file. -f OR -m or -j must be provided when creating a new SKU -j JSON_FILE, --json_file JSON_FILE SKU definition from config_db.json file. -f OR -m OR -j must be provided when creating a new SKU -pp PORT_SPLIT PORT_SPLIT, --port_split PORT_SPLIT PORT_SPLIT port name and split -b BASE, --base BASE SKU base definition -r, --remove Remove SKU folder -k HWSKU, --hwsku HWSKU SKU name to be used when creating a new SKU or for L2 configuration mode -p, --print Print port_config.ini without creating a new SKU -vv, --verbose Verbose output

**- What I did** I added a new script file sonic_sku_create.py to generate a new HWSKU based on XML file or Minigraph. But, focus is on creating HWSKU based on XML file and not on the minigraph and l2 mode. I also added a unit test pyTest script sku_create_test.py in sonic-utilities-test folder to test the script in Unit Testing mode. Motivation: To create SKU for Mellanox platforms based on XML file with Port related inputs and also through Minigraph file. This tool also allows to split a port or unsplit ports based on configuration which modifies the port related information in port_config.ini and config_db.json usage: sonic_sku_create.py [-h] [-v] (-f FILE | -m [MINIGRAPH_FILE] | -j JSON_FILE | -pp PORT_SPLIT PORT_SPLIT) [-b BASE] [-r] [-k HWSKU] [-p] [-vv] Create a new SKU optional arguments: -h, --help show this help message and exit -v, --version show program's version number and exit -f FILE, --file FILE SKU definition from xml file. -f OR -m or -j must be provided when creating a new SKU -m [MINIGRAPH_FILE], --minigraph_file [MINIGRAPH_FILE] SKU definition from minigraph file. -f OR -m or -j must be provided when creating a new SKU -j JSON_FILE, --json_file JSON_FILE SKU definition from config_db.json file. -f OR -m OR -j must be provided when creating a new SKU -pp PORT_SPLIT PORT_SPLIT, --port_split PORT_SPLIT PORT_SPLIT port name and split -b BASE, --base BASE SKU base definition -r, --remove Remove SKU folder -k HWSKU, --hwsku HWSKU SKU name to be used when creating a new SKU or for L2 configuration mode -p, --print Print port_config.ini without creating a new SKU -vv, --verbose Verbose output **- How I did it** I did it by adding these files which are required for new SKU generation in the forthcoming releases. **- How to verify it** I verified all the options of sonic_sku_create.py file and tested them thoroughly as well as created a test case in sku_create_test.py file to test the functionality of sonic_sku_create.py in Unit Test mode.

ccb5245 (HEAD -> 201911, origin/201911) [fast-reboot] Fix fast-reboot when NDP entries are present (sonic-net#1295) d09667b Multi-ASIC support for show ip(v6) route (201911 branch) (sonic-net#1283) 28399bf [201911-Mellanox] SKU creator Tool (sonic-net#1163) (sonic-net#1250) Signed-off-by: Abhishek Dosi <[email protected]>

Madhan Babu added 7 commits October 13, 2020 18:54

Added file sonic_sky_create.py script to sonic-utilities/scripts to c…

2c239ec

…reate a new HWSKU based on XML file. Also added file sku_create_test.py pyTest unit test script to test the same

address enhancements suggested by the team

e01828f

Addressed code review comments

2afbf31

Fixes due to more testing in 201911 image

983d3dc

fixed indentation errors in the script

8f5a276

fixing LGTM warnings

12572ca

fixed redundant empty lines

58e9708

madhanmellanox force-pushed the madhan_skucreate branch from 763db24 to 58e9708 Compare October 13, 2020 15:57

added @classmethod to tear_down() member function

b896c93

qiluo-msft reviewed Oct 20, 2020

View reviewed changes

Addressing code review comments

6926bd9

qiluo-msft reviewed Nov 4, 2020

View reviewed changes

Address second set of code review comments from Qi

9eb7121

Madhan Babu added 2 commits November 4, 2020 22:49

corrected the argument for verbose

40b36fe

Addressed LGTM comment

d5e5c8c

qiluo-msft approved these changes Nov 5, 2020

View reviewed changes

qiluo-msft merged commit 99de167 into sonic-net:master Nov 5, 2020

Conversation

madhanmellanox commented Oct 12, 2020

Uh oh!

lgtm-com bot commented Oct 12, 2020

Uh oh!

lgtm-com bot commented Oct 13, 2020

Uh oh!

madhanmellanox commented Oct 13, 2020

Uh oh!

madhanmellanox commented Oct 13, 2020

Uh oh!

madhanmellanox commented Oct 14, 2020

Uh oh!

madhanmellanox commented Oct 14, 2020

Uh oh!

madhanmellanox commented Oct 20, 2020

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

[](start = 8, length = 1)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Nov 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Nov 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lgtm-com bot commented Nov 4, 2020

Uh oh!

jleveque commented Nov 6, 2020

Uh oh!

madhanmellanox commented Nov 12, 2020

Uh oh!

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Nov 4, 2020 •

edited

Loading

qiluo-msft Nov 4, 2020 •

edited

Loading