Fix inconsistent mux state

[lolyu]

Description of PR

Summary:
Fixes # (issue)

Type of change

• Bug fix
• New feature
• Doc/Design
• Unit test

Approach

What is the motivation for this PR?

This issue occurs when running config load_minigraph to load new configs at both ToRs.
After write_standby.py, the muxorch will try to direct all traffic downstream to the SoC IP and server IP to the tunnel, but xcvrd might fail to set the adminforwardingstate of the port via gRPC because the gRPC channel could not be established because the other ToR is in standby state.
After linkmgrd starts to run and receives heartbeats from itself, it will try to toggle to active[toggle#1], but xcvrd might not be able to make the hardware toggle at the moment, so linkmgrd will mux wait. Also, because the mux probe table is initialized with Unknown state, linkmgrd will handle the initial mux probe state to have the composite states (active, unknown, up) and tries to probe the mux state.
As the muxorch has been toggled to active[toggle#1], the gRPC channel will be established at some point after, xcvrd will be able to answer the mux probe, so the linkmgrd will be able to change into (active, active, up) state.
But the toggling to active[toggle#1] is only finished half way, the mux status in STATE_DB:MUX_CABLE_TABLE is not updated, so show mux status will show unknown for those ports.

Signed-off-by: Longxiang Lyu lolv@microsoft.com

How did you do it?

When the linkmgrd changes into states (active, active, up) and has the original mux state as unknown, it will toggle the mux to active again to have those DB tables updated:

linkmgrd  -> APP_DB:MUX_CABLE_TABLE -> swss -> APP_DB:HW_MUX_CABLE_TABLE -> xcvrd 
xcvrd -> STATE_DB:HW_MUX_CABLE_TABLE -> swss -> STATE_DB:MUX_CABLE_TABLE -> linkmgrd  

How did you verify/test it?

On dualtor-mixed topo with icmp_responder running, do config load_minigraph on both ToRs, verify the show mux status on both ToRs:

root@demo-switch:~# show mux status
PORT        STATUS    HEALTH     HWSTATUS    LAST_SWITCHOVER_TIME
----------  --------  ---------  ----------  ---------------------------
Ethernet4   active    healthy    consistent
Ethernet8   active    healthy    consistent
Ethernet12  active    healthy    consistent
Ethernet16  active    healthy    consistent
Ethernet20  active    healthy    consistent
Ethernet24  active    healthy    consistent
Ethernet28  active    healthy    consistent
Ethernet32  active    healthy    consistent
Ethernet36  active    healthy    consistent
Ethernet40  active    healthy    consistent
Ethernet44  active    healthy    consistent
Ethernet48  active    healthy    consistent

Any platform specific information?

Documentation