Fix in config override when all asic namespaces not present in golden_config_db

[judyjoseph]

What I did

Fixes sonic-net/sonic-buildimage#16164

How I did it

In the multi-asic devices for eg: supervisor card of a chassis, the asics are not of its own -- it is the asic present in the fabric cards in the chassis. On then -- all the fabric cards need not be present in the chassis everytime.

In that case golden_config_db.json file generated by NDM ( based on the Subdevices in the minigraph ) will not have all ASICs -- it will be for the fabric cards present in the chassis

With the current logic, if there is an asic namespace missing -- config load_minigraph will exit with error as in the Issue #sonic-net/sonic-buildimage#16164 and not restart dockers.

Add logic to check and continue if the "asic" namespace is not present in the golden_config_db.json for multi-asic platforms.

How to verify it

Verified on a SUP

admin@str2-xxxx-sup-1:~$ sudo config load_minigraph -y --override_config 
Disabling container monitoring ...
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json   --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic0 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic1 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic2 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic3 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic4 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic5 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic6 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic7 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic8 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic9 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic10 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic11 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic12 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic13 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic14 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic15 --write-to-db
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Running command: config qos reload --no-dynamic-buffer --no-delay
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/0/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/1/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/2/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/3/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/4/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/5/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/6/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/7/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/8/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/9/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/10/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/11/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/12/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/13/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/14/buffers.json.j2
Buffer definition template not found at /usr/share/sonic/device/x86_64-nokia_ixrxxxxe_sup-r0/Nokia-IXRxxxxE-SUP-10/15/buffers.json.j2
Running command: pfcwd start_default
Running command: config override-config-table /etc/sonic/golden_config_db.json
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Override config not present for asic1                                   <<<<<<<<<<<<<<<<<<<<<
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Removing configDB overriden table first ...
Overriding input config to configDB ...
Overriding completed. No service is restarted.
Override config not present for asic14                          <<<<<<<<<<<<<<<<<<<<<
Override config not present for asic15                          <<<<<<<<<<<<<<<<<<<<<
Restarting SONiC target ...
Enabling container monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

[wenyiz2021]

hi @judyjoseph , please also change the UT return code to 0 as now we don't exit if asic is not found

self = <tests.config_override_test.TestConfigOverrideMultiasic object at 0x7fbd019a37f0>

def test_masic_missig_asic_override(self):
    def read_json_file_side_effect(filename):
        with open(MULTI_ASIC_MISSING_ASIC_OV, "r") as f:
            wrong_format = json.load(f)
        return wrong_format
    db = Db()
    cfgdb_clients = db.cfgdb_clients

    with mock.patch('config.main.read_json_file',
                    mock.MagicMock(side_effect=read_json_file_side_effect)):
        runner = CliRunner()
        result = runner.invoke(config.config.commands["override-config-table"],
                               ['golden_config_db.json'], obj=db)
        assert "Override config not present for asic1" in result.output
        # make sure program aborted with return code 1

      assert result.exit_code == 1

E assert 0 == 1
E +0
E -1

[wenyiz2021]

we may also need to change sys.exit(1) on line 1905

[wenyiz2021]

line 1905 and UT changed needed

[judyjoseph]

hi @judyjoseph , please also change the UT return code to 0 as now we don't exit if asic is not found

self = <tests.config_override_test.TestConfigOverrideMultiasic object at 0x7fbd019a37f0>

def test_masic_missig_asic_override(self):
    def read_json_file_side_effect(filename):
        with open(MULTI_ASIC_MISSING_ASIC_OV, "r") as f:
            wrong_format = json.load(f)
        return wrong_format
    db = Db()
    cfgdb_clients = db.cfgdb_clients

    with mock.patch('config.main.read_json_file',
                    mock.MagicMock(side_effect=read_json_file_side_effect)):
        runner = CliRunner()
        result = runner.invoke(config.config.commands["override-config-table"],
                               ['golden_config_db.json'], obj=db)
        assert "Override config not present for asic1" in result.output
        # make sure program aborted with return code 1

      assert result.exit_code == 1

E assert 0 == 1 E +0 E -1

Done

[judyjoseph]

we may also need to change sys.exit(1) on line 1905

We will keep the exit(1) here as the "localhost" which relates to DEFAULT_NAMESPACE (linux host ) should be present

[judyjoseph]

@qiluo-msft @wen587 please review, this is currently breaking config load_minigraph in a chassis where fabric cards are not fully populated (which is usually the case)

[gechiang]

@judyjoseph , Is this fix also applicable to 3164 HWSKU which is also multi-Asic? This will decide if the fix should go to public 202205 or to MSF repo 202205 .

[gechiang]

@judyjoseph Please provide the MSFT ADO number for this fix.

[qiluo-msft]

/azp run Azure.sonic-utilities

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[gechiang]

@judyjoseph do we also need backport for 202305 release as well?

[judyjoseph]

@judyjoseph do we also need backport for 202305 release as well?

yes just added the label - Thanks @gechiang. This scenario is not significant to the other platform as it is fixed ASIC configuration.

[gechiang]

In the interest of time, I went ahead and created the MSFT ADO: 24914487
@yxieca , @StormLiangMS , please help cherry-pick (backport) this fix to corresponding releases.

[gechiang]

@yxieca can you also help approve this for 202211? Thanks!