retry the chassis db cleanup operations

[arlakshm]

Why I did it

When running load_minigraph or reloading configuration on the linecards, the interface-config.service restarts, which causes the midplane interface to flap. If swss.sh on the linecards deletes state from chassis_db, some states may not be cleaned up correctly, while others are successfully removed. For example, cleanup for SYSTEM_NEIGHBOR or SYSTEM_INTF may fail, but SYSTEM_LAG cleanup might succeed. This can lead to inconsistent lag IDs for the remote LC.

Work item tracking

• Microsoft ADO 35454463

How I did it

Add logic to retry in swss.sh script.

How to verify it

Run test to do load_minigraph on all the linecards and check for the logs to for remove lag failure for Lags on remote LC.

Which release branch to backport (provide reason below if selected)

• 202205
• 202211
• 202305
• 202311
• 202405
• 202411
• 202505

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[arlakshm]

@saksarav-nokia @ysmanman, @mlok-nokia, @arista-nwolfe, please help review...

[saksarav-nokia]

@arlakshm , The code changes LGTM. However, i am unable to understand when we could hit this issue. If IMM can't reach the CHASSIS_APP_DB, then the following code waits till it is reachable right?
until [[ $($SONIC_DB_CLI CHASSIS_APP_DB PING | grep -c True) -gt 0 ]]; do
sleep 1
done

Also we fixed something in this area while ago with PR https://github.com/sonic-net/sonic-buildimage/pull/18756

[deepak-singhal0408]

LGTM.

[arlakshm]

@arlakshm , The code changes LGTM. However, i am unable to understand when we could hit this issue. If IMM can't reach the CHASSIS_APP_DB, then the following code waits till it is reachable right? until [[ $($SONIC_DB_CLI CHASSIS_APP_DB PING | grep -c True) -gt 0 ]]; do sleep 1 done

Also we fixed something in this area while ago with PR https://github.com/sonic-net/sonic-buildimage/pull/18756

Hi @saksarav-nokia, during load_minigraph the networking service is restarted, which happens parallely so the midplane interface can flap after this code is executed $($SONIC_DB_CLI CHASSIS_APP_DB PING | grep -c True) -gt 0 ]]; do sleep 1 done. In this case some the clean up in the chassis_db is not done. This PR to fix this case

[mssonicbld]

Cherry-pick PR to msft-202405: Azure/sonic-buildimage-msft#1730