[Chassis] Bug: test_pretest fails after FRR upgrade to 10.3 from 10.0.1

[ymd-arista]

Is it platform specific

generic

Importance or Severity

Critical

Description of the bug

In our runs, on master branch, we are seeing test_pretest.py failures.

In the logs, we see that route_check.py fails and causes monit to fail aswell.

2025 May 16 15:41:18.285938 nfc420-8 WARNING route_check.py: Failure results: {{#012    "": {#012        "missed_ROUTE_TABLE_routes": [#012            "127.100.0.0/16"#012        ]#012    }#012}}
2025 May 16 15:41:18.286005 nfc420-8 WARNING route_check.py: Failed. Look at reported mismatches above
2025 May 16 15:41:18.286033 nfc420-8 WARNING route_check.py: add: {#012    "": []#012}
2025 May 16 15:41:18.286058 nfc420-8 WARNING route_check.py: del: {#012    "": []#012}
...
2025 May 16 15:42:17.061130 nfc420-8 ERR monit[868]: 'routeCheck' status failed (255) -- Failure results: {{#012    "": {#012        "missed_ROUTE_TABLE_routes": [#012            "127.100.0.0/16"#012        ]#012    }#012}}#012Failed. Look at reported mismatches above#012add: {#012    "": []#012}#012del: {#012    "": []#012}

We noticed that show ip route check 127.100.0.0 returns different output on the image where pretest was passing compared to latest version.

On latest version:

root@nfc420-7:~# show ip route 127.100.0.0
Routing entry for 127.100.0.0/16
  Known via "kernel", distance 0, metric 0, best
  Last update 14:35:01 ago
  * directly connected, eth1-midplane

On previous version, where pretest was passing:

root@nfc406-7:~# show ip route 127.100.0.0
Routing entry for 0.0.0.0/0
  Known via "bgp", distance 20, metric 0, best
  Last update 00:20:56 ago
  * 10.0.0.1, via PortChannel2002
  * 10.0.0.5, via Ethernet16
  * 10.0.0.7, via PortChannel2006
  * 10.0.0.11, via Ethernet28

We weren't sure where this route is coming from K>*127.100.0.0/16 [0/0] is directly connected, eth1-midplane, 18:42:42.

But, we noticed that both have different FRR versions.

nfc420-7# show version
FRRouting 10.3 (nfc420-7) on Linux(6.1.0-22-2-amd64).

nfc406-7# show version
FRRouting 10.0.1 (nfc406-7) on Linux(6.1.0-22-2-amd64).

Reverting FRR 10.3 to 10.0.1 seems to have solved this issue.

Steps to Reproduce

Load the latest master image and run pretest with it.

Actual Behavior and Expected Behavior

Expected Behavior: test_pretest.py passes

Actual Behavior: test_pretest.py errors out.

Relevant log output

Output of show version

Attach files (if any)

No response

[arista-nwolfe]

FRR bump up PR: sonic-net/sonic-buildimage#22267

[abdosi]

@anamehra @mannytaheri please check and comment

[arista-nwolfe]

@anamehra @mannytaheri please check and comment

@anamehra @mannytaheri did you get a chance to take a look at this issue?
It's currently blocking us from running sonic-mgmt on chassis in master.

[mannytaheri]

I did a rebase on master today and ran test_pretest.py. I don't see the issue on Nokia chassis.

[mannytaheri]

We don't see the issue at Nokia with the latest image and the latest master sonic-mgmt.

[arista-nwolfe]

It looks like this type of issue (new routes showing up after a FRR bump up) has happened before.
Issue: sonic-net/sonic-buildimage#6483
We saw routes showing up in APPL_DB for 0.0.0.0 after bump up from 7.2 to 7.3.

It was fixed by ignoring routes that pointed to eth0 or docker0 in fpmsyncd
sonic-net/sonic-swss#1606

I wonder if we need to do the same thing but for eth1-midplane now.

[ahsalam]

@ymd-arista, @arista-nwolfe
I think we need to change the issue title. The issue is not related to the FRR Upgrade.

[arista-nwolfe]

@ymd-arista, @arista-nwolfe I think we need to change the issue title. The issue is not related to the FRR Upgrade.

Why do you say it isn't related to the FRR Upgrade?
We hit this problem with FRR10.3 but we don't hit this problem with FRR10.0.1.