[arp_update]: Fix IPv6 neighbor race condition

[theasianpianist]

Why I did it

A race condition exists which makes it possible for the kernel to resolve a trapped packet's destination IP at the same time that arp_update is running the ip neigh replace command for that neighbor IP. When this occurs, the kernel's neighbor entry for this IP is in the INCOMPLETE state, and the ip neigh replace command sets it to permanently incomplete. This means no netlink message will be generated for this neighbor, since the kernel doesn't generate netlink messages for INCOMPLETE neighbors (it would only generate a message once the neighbor transitions to FAILED, which doesn't happen due to the ip neigh replace command). As a result, no APPL_DB neighbor table entry is ever created and no tunnel route for the IP is ever installed, leading to dropped traffic.

Work item tracking

• Microsoft ADO (number only): 24152065

How I did it

• Check each FAILED or INCOMPLETE neighbor to see if a corresponding APPL_DB neighbor table entry exists. If no entry exists, flush the neighbor from the kernel
• After pinging the neighbor IPs, wait for any neighbor entries which might be transiently INCOMPLETE to transition to FAILED (so that the subsequent ip neigh replace command can set them to permanently incomplete)
• Exclude any INCOMPLETE neighbors from the ip neigh replace command in case they are new neighbors for which no netlink message has been generated yet

How to verify it

• Run arp_update with no FAILED or INCOMPLETE neighbor entries, verify no changes are made to the kernel neighbor table

• Run arp_update with a FAILED neighbor entry with corresponding APPL_DB entry, verify the neighbor IP is pinged and set to INCOMPLETE permanently

• Run arp_update with an INCOMPLETE neighbor entry with corresponding APPL_DB entry, verify the neighbor IP is pinged and set to INCOMPLETE permanently

• Run arp_update with a FAILED neighbor entry without corresponding APPL_DB entry, verify the neighbor is flushed, pinged, and set to INCOMPLETE permanently

• Run arp_update with an INCOMPLETE neighbor entry without corresponding APPL_DB entry, verify the neighbor is flushed, pinged, and set to INCOMPLETE permanently

• Run arp_update with various combinations of above scenarios - verify that only neighbors missing APPL_DB entries are flushed from the kernel; verify that all FAILED/INCOMPLETE neighbors are pinged and set to permanently INCOMPLETE

Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006
• 202012
• 202106
• 202111
• 202205
• 202211
• 202305

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

[prsunny]

@theasianpianist , can you please add Requestfor labels and remove the Approvedfor label. Approvedfor shall be done by branch owners

[prsunny]

can you please test this with traffic which can resolve a neighbor during a ping and another where neighbor cannot be resolved?

[theasianpianist]

Tested with one resolvable IP, one unresolvable IP, and one unresolvable IP where I manually deleted the APPL_DB entry. The script behaved as expected - the deleted IP was flushed, all three IPs were pinged, and the final state of the IPs was REACHABLE (for the resolvable IP) and INCOMPLETE for the other two.

[mssonicbld]

Cherry-pick PR to 202211: #15693

[mssonicbld]

Cherry-pick PR to 202205: #15694

[mssonicbld]

Cherry-pick PR to 202305: #15877