Fix route_check.py to not hog a lot of memory

[venkit-nexthop]

What I did

The route_check.py script launches a sonic "show ip route json" to detect any missing routes between FRR and app-db. This causes the show python program to hog a lot of memory (in one test I have noticed it went to 15G), causing the system to OOM. The issue is due to the fact that the show script is reading all the json content (using vtysh) in one shot and is attempting to interpret the json (using json.loads()).

How I did it

This diff modifies the route_check.py to not
invoke "show" and rather invoke the vtysh cmd directly. It then attempt to interpret one route at
a time in a paginated manner. This prevents a sudden transient memory buildup. The zebra process already does the right thing and backs off when the output socket buffers are full. There is probably scope to improve that further
(Refer to
https://sonicfoundation.dev/2025-sonic-hackathon-most-impactful-award-spotlight-optimizing-output-buffer-memory-for-show-commands/)

How to verify it

Earlier with a very high route-scale (or path scale when ECMP was high), route_check.py used to hog a lot of memory even causing OOMs. With this fix, the same route-scale is operating just fine.

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[venkit-nexthop]

@deepak-singhal0408
@prsunny
@eddieruan-alibaba

As discussed in the PM routing WG meeting on Jan 15 2026. Here is the PR for handling memory hog coming from route-check.py

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[deepak-singhal0408]

LGTM.. Thanks,

[venkit-nexthop]

@prsunny could you help me merge this.
@deepak-singhal0408 has approved the change.

[venkit-nexthop]

@deepak-singhal0408 this has not merged yet after your approval. Does it need some other maintainer to approve as well?

[deepak-singhal0408]

@rlhui @prsunny please help with the merge? thanks,

[rlhui]

@deepak-singhal0408 , does msft-202405 need this?

[deepak-singhal0408]

@deepak-singhal0408 , does msft-202405 need this?

no.. in msft-202405, frr_pending_route_check is (under bgp_supress_fib_pending flag which is disabled)..
https://github.com/Azure/sonic-utilities.msft/blob/202405/scripts/route_check.py#L867

[deepak-singhal0408]

@venkit-nexthop , the PR has cherry-pcik conflict for 202511.. please help create another PR?

[saiarcot895]

@venkit-nexthop This is causing the submodule update PR to fail KVM tests; the route_check.py script is failing to run. Is there a corresponding buildimage PR to fix this?

2026 Feb 6 06:35:18.571503 vlab-01 ERR monit[876]: 'routeCheck' status failed (1) -- Traceback (most recent call last):#012 File "/usr/local/bin/route_check.py", line 40, in <module>#012 import ijson#012 File "/usr/local/lib/python3.13/dist-packages/ijson/__init__.py", line 36, in <module>#012 backend = _default_backend()#012 File "/usr/local/lib/python3.13/dist-packages/ijson/__init__.py", line 32, in _default_backend#012 return get_backend(backend)#012 File "/usr/local/lib/python3.13/dist-packages/ijson/__init__.py", line 24, in get_backend#012 return importlib.import_module('ijson.backends.' + backend)#012 ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#012 File "/usr/lib/python3.13/importlib/__init__.py", line 88, in import_module#012 return _bootstrap._gcd_import(name[level:], package, level)#012 ~~~~~~~~~~~~~~~~~~~~~~^^^^^^

[venkit-nexthop]

@venkit-nexthop This is causing the submodule update PR to fail KVM tests; the route_check.py script is failing to run. Is there a corresponding buildimage PR to fix this?

2026 Feb 6 06:35:18.571503 vlab-01 ERR monit[876]: 'routeCheck' status failed (1) -- Traceback (most recent call last):#012 File "/usr/local/bin/route_check.py", line 40, in <module>#012 import ijson#012 File "/usr/local/lib/python3.13/dist-packages/ijson/__init__.py", line 36, in <module>#012 backend = _default_backend()#012 File "/usr/local/lib/python3.13/dist-packages/ijson/__init__.py", line 32, in _default_backend#012 return get_backend(backend)#012 File "/usr/local/lib/python3.13/dist-packages/ijson/__init__.py", line 24, in get_backend#012 return importlib.import_module('ijson.backends.' + backend)#012 ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#012 File "/usr/lib/python3.13/importlib/__init__.py", line 88, in import_module#012 return _bootstrap._gcd_import(name[level:], package, level)#012 ~~~~~~~~~~~~~~~~~~~~~~^^^^^^

Hi @saiarcot895
I was not aware of this breakage. Does the submodule merge process run some additional tests that are not usually covered when the original change is tested?
The failure appears to be due to the environment where it is running. It seems like on this environment ijson is not able to find the C-libraries required to set a default backend.
One way to fix this is to force a python backend.

The following single line change should fix it:

- import ijson
+ import ijson.backends.python as ijson

@deepak-singhal0408 recommended the use of this library. Deepak do you have any other suggestions/recommendations?

[saiarcot895]

sonic-utilities PRs just run unit tests that are located in this repository. However, when we do submodule updates in sonic-buildimage (and an image is built with this change), that includes sonic-mgmt tests running in a KVM environment, which is why the breakage doesn't appear here, but does appear there.

[venkit-nexthop]

sonic-utilities PRs just run unit tests that are located in this repository. However, when we do submodule updates in sonic-buildimage (and an image is built with this change), that includes sonic-mgmt tests running in a KVM environment, which is why the breakage doesn't appear here, but does appear there.

Thanks @saiarcot895 I will post a PR to fix this now.

[saiarcot895]

Hold off on that, there's something stranger going on.

[venkit-nexthop]

Hold off on that, there's something stranger going on.

Sure

[saiarcot895]

Ok, so monit specifically is failing to run route_check.py, because the systemd service file for monit sets MemoryDenyWriteExecute=true, meaning the process cannot create any memory mapping that sets both the write and execute permissions. However, when ijson is trying to use the yajl2 C backend via CFFI, CFFI tries to make a RWX memory mapping:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import ijson
  File "/usr/local/lib/python3.13/dist-packages/ijson/__init__.py", line 36, in <module>
    backend = _default_backend()
  File "/usr/local/lib/python3.13/dist-packages/ijson/__init__.py", line 32, in _default_backend
    return get_backend(backend)
  File "/usr/local/lib/python3.13/dist-packages/ijson/__init__.py", line 24, in get_backend
    return importlib.import_module('ijson.backends.' + backend)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/dist-packages/ijson/backends/yajl2_cffi.py", line 88, in <module>
    @ffi.callback('int(void *ctx)')
     ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/cffi/api.py", line 396, in callback_decorator_wrap
    return self._backend.callback(cdecl, python_callable,
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
                                  error, onerror)
                                  ^^^^^^^^^^^^^^^
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

As per the documentation, ijson would need to refactor their CFFI usage so that it doesn't use ffi.callback(). I don't know how much effort this requires on their part.

There are three workarounds:

1. Add an override file for monit to disable MemoryDenyWriteExecute. This opens a security hole, so I'd discourage it.
2. Add an override file to set the environment variable IJSON_BACKEND=yajl2. This uses ctypes instead of CFFI to talk to yajl2, and works without making a RWX memory mapping.
3. Use import ijson.backends.python as ijson (as you suggested). This will work as well, but might reduce in lower performance.

I personally prefer option 2, or maybe some hybrid of option 2 and 3 (try import ijson.backends.yajl2 as ijson, fall back to import ijson.backends.python as ijson if that fails), just to get the most performance possible.

[venkit-nexthop]

@saiarcot895 interesting....

When I stop the route_check.py right after the import using a pdb and check the /proc//maps, I dont find a "wx" entry at all. Perhaps, the page gets unmapped after the import?

In any case, I agree that overriding the systemd settings to allow for process to map pages with w+x is a security hole. I will do something like this:

try:
    import ijson.backends.yajl2 as ijson
except ImportError:
    import ijson.backends.python as ijson