[202405] [Chassis]: Ports take too long to come up due to delayed port up notification processing by orchagent

[mannytaheri]

Issue Description

The ports take more the 20 minutes to come up due to the delayed port up notification processing by orchagent after reload/reboot in T2 topo.

Results you see

The port up notifications are queued due to lot of bgp route (34000 routes) updates and take a long time.
This occurs after a config reload or a reboot.

Results you expected to see

The bgp routes update should be handled correctly and ports should come up in a reasonable time.

Is it platform specific

generic

Relevant log output

No response

Output of show version

admin@ixre-egl-board15:~$ show ver

SONiC Software Version: SONiC.HEAD.742851-nokia-master-ef3457c7
SONiC OS Version: 12
Distribution: Debian 12.5
Kernel: 6.1.0-11-2-amd64
Build commit: ef3457c7
Build date: Fri Jun 14 19:31:02 UTC 2024
Built by: gitlab-runner@sonic-build-server04

Attach files (if any)

No response

[arlakshm]

Hi @mannytaheri,
Can you please patch these 2 PR and see if the port comes up faster
sonic-net/sonic-host-services#135
#19482

[wenyiz2021]

@mannytaheri please let me know the test result with the patches, if not working I'll look at

[deepak-singhal0408]

PMON faster bring up does not seem to help this issue. @wenyiz2021 could you help follow up with Arvind/BRCM and try out the BRCM fix

[judyjoseph]

Related issues raised earlier ? : #17180

[judyjoseph]

To further debug

1. Why this issue is seen newly in master/202405 -- not in 202205
2. Is the ports coming up delayed/faster, coming around same time as BGP route updates
3. Any new dockers/systemd added causing dependency
4. eventd logs, can we disable evend docker ?

Check with sonic-common-infra subgroup, for a root cause which could be known.

[judyjoseph]

The FIB suppress pending feature got merged recently, can we check again with latest master.202405 build #19736 @mannytaheri

[wenyiz2021]

cannot reproduce issue on Arista chassis with latest master image with SAI 11.2 taken from #19854

[wenyiz2021]

@mannytaheri this seems not general issue for all platform? can you try above master image with SAI 11?

[abdosi]

[abdosi]

above is the understanding.