While generating configs through sonic-cfggen , two sets of "DEVICE_METADATA" and "PORT" are created and loading config fails .

[chitra-raghavan]

Description

While generating configs through sonic-cfggen , two sets of DEVICE_METADATA and PORT sets are created and loading config fails .
ASIC: TH3
SKU: DellEMC-Z9332f-M-O16C64

Steps to reproduce the issue:

1. sonic-cfggen -H -k DellEMC-Z9332f-M-O16C64 -H --preset l2 --print-data > config_db.json

Describe the results you received:

Config_db should get loaded without any errors

Describe the results you expected:

As two sets of DEVICE_METADATA" and "PORT" are created , loading config fails.
After removing the redundant set , it passed. Attached the wrong config_db for reference.

root@sonic-10429:~# config reload  config_db.json
Clear current config and reload config from the file(s) config_db.json ? [y/N]: y
Disabling container monitoring ...
Stopping SONiC target ...

Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j config_db.json --write-to-db
Traceback (most recent call last):
  File "/usr/local/bin/sonic-cfggen", line 431, in <module>
    main()
  File "/usr/local/bin/sonic-cfggen", line 326, in main
    _process_json(args, data)
  File "/usr/local/bin/sonic-cfggen", line 237, in _process_json
    deep_update(data, FormatConverter.to_deserialized(json.load(stream)))
  File "/usr/lib/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 504 column 1 (char 14087)

{
    "DEVICE_METADATA": {
        "localhost": {
            "hwsku": "DellEMC-Z9332f-M-O16C64",
            "platform": "x86_64-dellemc_z9332f_d1508-r0",
            "mac": "c8:f7:50:eb:c2:41"
        }
    },
    "PORT": {
        "Ethernet0": {
            "lanes": "33,34",
            "alias": "hundredGigE1/1/1",
            "index": "1",
            "speed": "100000"
        },
        "Ethernet2": {
            "lanes": "35,36",
            "alias": "hundredGigE1/1/2",
            "index": "1",
            "speed": "100000"
        },
...
...
omitted
---
       }
    }
}
{
    "DEVICE_METADATA": {
        "localhost": {
            "hwsku": "DellEMC-Z9332f-M-O16C64",
            "platform": "x86_64-dellemc_z9332f_d1508-r0",
            "mac": "c8:f7:50:eb:c2:41"
        }
    },
    "PORT": {
        "Ethernet0": {
            "lanes": "33,34",
            "alias": "hundredGigE1/1/1",
            "index": "1",
            "speed": "100000",
            "admin_status": "up"
        },
        "Ethernet2": {
            "lanes": "35,36",
            "alias": "hundredGigE1/1/2",
            "index": "1",
            "speed": "100000",
            "admin_status": "up"
        },

       }
    }
}

Output of show version:

root@sonic-10429:~# show ver

SONiC Software Version: SONiC.202012.21933-1f75c4fbd
Distribution: Debian 10.10
Kernel: 4.19.0-12-2-amd64
Build commit: 1f75c4fbd
Build date: Wed Jun 30 14:10:58 UTC 2021
Built by: AzDevOps@sonic-build-workers-000FOX

Platform: x86_64-dellemc_z9332f_d1508-r0
HwSKU: DellEMC-Z9332f-M-O16C64
ASIC: broadcom
ASIC Count: 1
Serial Number: TH04CN21CET009BR0023
Uptime: 05:07:48 up 1 day, 14:06,  1 user,  load average: 0.79, 0.72, 0.73

Docker images:
REPOSITORY                    TAG                      IMAGE ID            SIZE
docker-sonic-mgmt-framework   202012.21933-1f75c4fbd   dd058689c5b4        620MB
docker-sonic-mgmt-framework   latest                   dd058689c5b4        620MB
docker-sonic-telemetry        202012.21933-1f75c4fbd   ef340a501802        490MB
docker-sonic-telemetry        latest                   ef340a501802        490MB
docker-fpm-frr                202012.21933-1f75c4fbd   22da11c254a0        429MB
docker-fpm-frr                latest                   22da11c254a0        429MB
docker-nat                    202012.21933-1f75c4fbd   bdf3d0e355a1        414MB
docker-nat                    latest                   bdf3d0e355a1        414MB
docker-orchagent              202012.21933-1f75c4fbd   d491b29f1642        430MB
docker-orchagent              latest                   d491b29f1642        430MB
docker-sflow                  202012.21933-1f75c4fbd   41109217e123        412MB
docker-sflow                  latest                   41109217e123        412MB
docker-platform-monitor       202012.21933-1f75c4fbd   d247c6c20279        608MB
docker-platform-monitor       latest                   d247c6c20279        608MB
docker-teamd                  202012.21933-1f75c4fbd   6b5e297a5815        411MB
docker-teamd                  latest                   6b5e297a5815        411MB
docker-syncd-brcm             202012.21933-1f75c4fbd   c6ed0860eff1        693MB
docker-syncd-brcm             latest                   c6ed0860eff1        693MB
docker-snmp                   202012.21933-1f75c4fbd   3ee0c6c29a84        442MB
docker-snmp                   latest                   3ee0c6c29a84        442MB
docker-lldp                   202012.21933-1f75c4fbd   1238ff90709f        441MB
docker-lldp                   latest                   1238ff90709f        441MB
docker-database               202012.21933-1f75c4fbd   ffca3e89f3e1        401MB
docker-database               latest                   ffca3e89f3e1        401MB
docker-router-advertiser      202012.21933-1f75c4fbd   091bd0e7101b        401MB
docker-router-advertiser      latest                   091bd0e7101b        401MB
docker-dhcp-relay             202012.21933-1f75c4fbd   1241fee1a43e        408MB
docker-dhcp-relay             latest                   1241fee1a43e        408MB

root@sonic-10429:~#

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

config_db.txt

[zhangyanzhao]

Looks like sonic-cfggen issue. @chitra-raghavan will check other platforms and other versions, update the thread. @gechiang please take a look

[gechiang]

I was able to reproduce this issue as well over a S6100 DUT. So this is not a specific issue with TH3.
It appears that the option "--preset l2" is already including the META_DATA and PORT Configuration while the hwsku option also included the META_DATA and PORT already... Will investigate deeper to see if this is broken recently or the bug has been there for a while...

[gechiang]

@lguohan
The root cause is somewhat caused by the following PR as well as user usage error...
#4941
Which allowed both options "--preset l2" and "--print-data" to be specified as parameters at the same execution iteration. This was not allowed previously and both were set to mutually exclusive of each other. If both are allowed we see this duplication issue together with the user redirected the output to a file... Can @tahmed-dev please comment on why these two options are no longer mutually exclusive of each other any more?

As for the usage error:
From the code It looks to me that the option "--print-data" will print what is currently present in "data". I believe the original intention of this parameter is to print it to the output screen, However the output is being redirected to "> config_db.json" and the processing of "--preset l2" kicks in afterwards which then again print the "data" again after it added the L2 preset configs.... This results to duplication where the extra info was due to the "--print-data".

So question to @chitra-raghavan What is the purpose of specifying both "--print-data" as well as redirecting the output to a file? I believe if you want to redirect the output to a file you should not specify "--print-data". you will see two set of output redirected to your file.

If you issue the following instead you should not observe any duplications:
sonic-cfggen -H -k DellEMC-Z9332f-M-O16C64 -H --preset l2 > config_db.json
Also, there is no need to specify -H twice although it did not cause any implications...

If you feel this is acceptable response please go ahead accept and close this issue.
Thanks!

[chitra-raghavan]

@gechiang , thanks will close this issue