saidump on T2 results lua script to take more than 5 sec

[abdosi]

On T2 with more route scale we see LUA script: https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua invoked as part of saidump commands can take more than 5 sec and thus all other process start failing as no/incorrect reply from Redis.

Command
generate_dump -s yesterday

Jan 30 23:22:32.637122 str2-xxxx-lc1-3 INFO acms#start.py: start: main: Waiting for bootstrap cert
Jan 30 23:23:00.293090 str2-xxxx-lc1-3 DEBUG syncd0#saidump: :> main: enter
Jan 30 23:23:00.307986 str2-xxxx-lc1-3 INFO syncd0#saidump: :- loadRedisScript: lua script local keys = redis.call("KEYS", KEYS[1] .. ":*")#012local res = {}#012#012for i,k in pairs(keys) do#012   local sres={}#012#012   local flat_map = redis.call('HGETALL', k)#012   for j = 1, #flat_map, 2 do#012       sres[flat_map[j]] = flat_map[j + 1]#012   end#012#012   res[k] = sres#012end#012#012return cjson.encode(res)#012 loaded, sha: 654245aafba722f7b601e1fc414c3647058c053c
Jan 30 23:23:05.329309 str2-xxxx-lc1-3 INFO database0#supervisord: redis 39:M 30 Jan 2023 23:23:05.328 # Lua slow script detected: still in execution after 5021 milliseconds. You can try killing the script using the SCRIPT KILL command. Script SHA1 is: 654245aafba722f7b601e1fc414c3647058c053c
Jan 30 23:23:05.329391 str2-xxxxx-lc1-3 ERR gbsyncd0#GBSAI[18]: :- checkReplyType: Expected to get redis type 5 got type 6, err: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.

[abdosi]

cc @judyjoseph @arlakshm @anamehra for viz.

[zhenggen-xu]

@abdosi do you have a sense how many routes would be too much? We could hit the issue in other layers too if this is related to route scale.

[gechiang]

@kcudnik please help investigate this. Thanks!