Warm reboot: Support portsyncd process warm restart

[jipanyang]

Signed-off-by: Jipan Yang [email protected]

What I did
Add support for portsyncd process warm restart.

Why I did it

How I verified it
Check warm restart count of the swss processes

root@sonic:/home/admin# redis-cli  
127.0.0.1:6379> keys WAR*
1) "WARM_RESTART_TABLE:portsyncd"
2) "WARM_RESTART_TABLE:orchagent"
3) "WARM_RESTART_TABLE:vlanmgrd"
4) "WARM_RESTART_TABLE:neighsyncd"
127.0.0.1:6379> hgetall  "WARM_RESTART_TABLE:orchagent"
1) "restart_count"
2) "1"
3) "state_restored"
4) "true"
127.0.0.1:6379> hgetall  "WARM_RESTART_TABLE:portsyncd"
1) "restart_count"
2) "1"
127.0.0.1:6379> 

Kill portsyncd process and start it again

admin@sonic:~$ docker exec -it swss /bin/bash
root@sonic:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.3  0.2  56476 16816 ?        Ss+  01:41   0:00 /usr/bin/python /usr/bin/supervisord
root        50  0.0  0.0 258684  3204 ?        Sl   01:41   0:00 /usr/sbin/rsyslogd -n
root        64  3.6  0.2 192820 16956 ?        Sl   01:41   0:03 /usr/bin/orchagent -d /var/log/swss -b 8192 -m 00:05:64:30:73:c0
root        76  1.4  0.0  99804  3908 ?        Sl   01:41   0:01 /usr/bin/portsyncd -p /usr/share/sonic/hwsku/port_config.ini
root        89  0.0  0.0  99864  3248 ?        Sl   01:41   0:00 /usr/bin/intfsyncd
root        92  0.0  0.0  99896  3788 ?        Sl   01:41   0:00 /usr/bin/neighsyncd
root       113  0.0  0.0  99876  3840 ?        Sl   01:41   0:00 /usr/bin/vlanmgrd
root       120  0.0  0.0  99820  4036 ?        Sl   01:41   0:00 /usr/bin/intfmgrd
root       131  0.0  0.0  99852  3760 ?        Sl   01:41   0:00 /usr/bin/buffermgrd -l /usr/share/sonic/hwsku/pg_profile_lookup.ini
root       146  0.0  0.0  20048  2884 ?        S    01:41   0:00 bash -c /usr/bin/arp_update; sleep 300
root       157  0.0  0.0   4236   712 ?        S    01:41   0:00 sleep 300
root       423  1.0  0.0  20244  3092 ?        Ss   01:43   0:00 /bin/bash
root       429  0.0  0.0  17504  2000 ?        R+   01:43   0:00 ps aux
root@sonic:/# pkill -x portsyncd
root@sonic:/# su 
su             sulogin        sum            supervisorctl  supervisord    suspend        
root@sonic:/# supervisorctl start portsyncd
portsyncd: started
root@sonic:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.2  0.2  56476 16852 ?        Ss+  01:41   0:00 /usr/bin/python /usr/bin/supervisord
root        50  0.0  0.0 258684  3204 ?        Sl   01:41   0:00 /usr/sbin/rsyslogd -n
root        64  2.4  0.2 192820 16956 ?        Sl   01:41   0:03 /usr/bin/orchagent -d /var/log/swss -b 8192 -m 00:05:64:30:73:c0
root        89  0.0  0.0  99864  3248 ?        Sl   01:41   0:00 /usr/bin/intfsyncd
root        92  0.0  0.0  99896  3788 ?        Sl   01:41   0:00 /usr/bin/neighsyncd
root       113  0.0  0.0  99876  3840 ?        Sl   01:41   0:00 /usr/bin/vlanmgrd
root       120  0.0  0.0  99820  4036 ?        Sl   01:41   0:00 /usr/bin/intfmgrd
root       131  0.0  0.0  99852  3760 ?        Sl   01:41   0:00 /usr/bin/buffermgrd -l /usr/share/sonic/hwsku/pg_profile_lookup.ini
root       146  0.0  0.0  20048  2884 ?        S    01:41   0:00 bash -c /usr/bin/arp_update; sleep 300
root       157  0.0  0.0   4236   712 ?        S    01:41   0:00 sleep 300
root       423  0.0  0.0  20244  3268 ?        Ss   01:43   0:00 /bin/bash
root       565  1.6  0.0  99804  3936 ?        Sl   01:43   0:00 /usr/bin/portsyncd -p /usr/share/sonic/hwsku/port_config.ini
root       585  0.0  0.0  17504  2180 ?        R+   01:43   0:00 ps aux

No traffic loss, also check restart count again, portsyncd restart_count incremented by 1

127.0.0.1:6379> 
127.0.0.1:6379> hgetall  "WARM_RESTART_TABLE:portsyncd"
1) "restart_count"
2) "2"
127.0.0.1:6379> hgetall  "WARM_RESTART_TABLE:orchagent"
1) "restart_count"
2) "1"
3) "state_restored"
4) "true"

Details if related
Has dependency on

sonic-net/sonic-swss-common#211
#547

[lguohan]

based on our discussion, we need to vs test for the warm reboot work flow. please add.

[jipanyang]

jipan@sonic-build-2:~/warm_reboot/sonic-buildimage/src/sonic-swss/tests$ sudo pytest -v --dvsname=vs test_warm_reboot.py
======================================================================= test session starts =======================================================================
platform linux2 -- Python 2.7.12, pytest-3.3.0, py-1.5.4, pluggy-0.6.0 -- /usr/bin/python
cachedir: .cache
rootdir: /home/jipan/warm_reboot/sonic-buildimage/src/sonic-swss/tests, inifile:
collected 1 item

test_warm_reboot.py::test_PortSyncdWarmRestart PASSED [100%]

==================================================================== 1 passed in 36.92 seconds ====================================================================

[lguohan]

retest this please

[qiluo-msft]

checkPortInitDone [](start = 5, length = 17)

I think it is not necessary to check port table. For warm reboot, ignore all code related initialization, and just keep processing netlink messages or port config events.

Maybe we could force processing all oper_status for warm reboot, so downstream orchagent will get latest oper status immediately after finishing its warm start. #Closed

[jipanyang]

Yes, the check is optional. It was designed to wok with the two different approaches, coupled/decoupled portsyncd and portorch. Will remove it.

[lguohan]

can you resolve conflict?

[lguohan]