Dev sysready extension by fastiuk · Pull Request #13 · fastiuk/sonic-buildimage

fastiuk · 2024-02-05T21:03:53Z

Why I did it

Work item tracking

Microsoft ADO (number only):

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

fastiuk · 2024-02-07T18:52:00Z

files/build_templates/init_cfg.json.j2

+{%- if feature in ["bgp"] %}
            "check_up_status" : "false",
 {%- endif %}
+{%- if feature in ["ib-utils", "snmp"] %}


Remove ib-utils

Please remove ib-utils as you commented

fastiuk · 2024-02-07T19:05:07Z

src/system-health/tests/test_system_health.py

    sysmon = Sysmonitor()
-    sysmon.publish_system_status('UP')
+    sysmon.monitor_timeout = MagicMock()
+    # sysmon.publish_system_status('UP')


fastiuk · 2024-02-07T19:06:21Z

src/system-health/tests/test_system_health.py

        assert call_args in expected_calls

-@patch('health_checker.sysmonitor.Sysmonitor.print_console_message', MagicMock())
+# @patch('health_checker.sysmonitor.Sysmonitor.post_system_status', MagicMock())


dgsudharsan · 2024-02-14T05:08:23Z

files/build_templates/docker_image_ctl.j2

+    # from DB. Read timeout from config file and add two extra minutes on top of
+    # it.
+    CONFFILE=system_health_monitoring_config.json
+    PLATFORM=$(sonic-cfggen -d -v "DEVICE_METADATA['localhost']['platform']")


Try to use cache rather than using sonic-cfggen as sonic-cfggen is a costly operation.

Please refer to https://github.com/sonic-net/sonic-buildimage/pull/17343

dgsudharsan · 2024-02-14T05:11:08Z

files/build_templates/docker_image_ctl.j2

+        TIMEOUT=10
+    fi
+
+    # Add to extra minutes and convert to seconds and


What is the rationale behind adding extra two minutes?

dgsudharsan · 2024-02-14T05:12:48Z

files/build_templates/docker_image_ctl.j2

    fi
 {%- elif docker_container_name == "snmp" %}
    $SONIC_DB_CLI STATE_DB HSET 'DEVICE_METADATA|localhost' chassis_serial_number $(decode-syseeprom -s)
+


Are you enabling this feature by default for snmp? Shouldn't it be based on configuration?

files/build_templates/init_cfg.json.j2

dgsudharsan · 2024-02-14T05:16:06Z

files/build_templates/init_cfg.json.j2

+{%- if feature in ["bgp"] %}
            "check_up_status" : "false",
 {%- endif %}
+{%- if feature in ["ib-utils", "snmp"] %}


Please remove ib-utils as you commented

dgsudharsan · 2024-02-14T05:17:38Z

files/scripts/syncd.sh

        /usr/bin/mlnx-fw-upgrade.sh -v
        if [[ "$?" -ne "${EXIT_SUCCESS}" ]]; then
            debug "Failed to upgrade fw. " "$?" "Restart syncd"
+            sonic-db-cli STATE_DB HSET "FEATURE|$DEV_SRV" fail_reason \


Why are we considering just the asic firmware update as sysready indication for syncd. Shouldn't it be the create switch success?

dgsudharsan · 2024-02-14T05:21:15Z

src/system-health/health_checker/sysmonitor.py

 REDIS_TIMEOUT_MS = 0
-system_allsrv_state = "DOWN"
-spl_srv_list = ['database-chassis', 'gbsyncd']
+spl_srv_list = ['database-chassis', 'gbsyncd', 'e2scrub_reap']


Can you clarify what is e2scrub_reap?

dgsudharsan · 2024-02-14T05:27:47Z

src/system-health/health_checker/sysmonitor.py

+
+# Subprocess to monitor system ready timeout. If timeout will be exceeded,
+# send a message to queue and exit
+class MonitorTimeout(ProcessTaskBase):


This thread needs to be spawned only when feature is enabled.

…et#21095) Adding the below fix from FRR FRRouting/frr#17297 This is to fix the following crash which is a statistical issue [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M dplane_fpm_nl -M snmp'. Program terminated with signal SIGABRT, Aborted. #0 0x00007fccd7351e2c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [Current thread is 1 (Thread 0x7fccd6faf7c0 (LWP 36))] (gdb) bt #0 0x00007fccd7351e2c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fccd7302fb2 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007fccd72ed472 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007fccd75bb3a9 in _zlog_assert_failed (xref=xref@entry=0x7fccd7652380 <_xref.16>, extra=extra@entry=0x0) at ../lib/zlog.c:678 #4 0x00007fccd759b2fe in route_node_delete (node=<optimized out>) at ../lib/table.c:352 #5 0x00007fccd759b445 in route_unlock_node (node=0x0) at ../lib/table.h:258 #6 route_next (node=<optimized out>) at ../lib/table.c:436 #7 route_next (node=node@entry=0x56029d89e560) at ../lib/table.c:410 #8 0x000056029b6b6b7a in if_lookup_by_name_per_ns (ns=ns@entry=0x56029d873d90, ifname=ifname@entry=0x7fccc0029340 "PortChannel1020") at ../zebra/interface.c:312 #9 0x000056029b6b8b36 in zebra_if_dplane_ifp_handling (ctx=0x7fccc0029310) at ../zebra/interface.c:1867 #10 zebra_if_dplane_result (ctx=0x7fccc0029310) at ../zebra/interface.c:2221 #11 0x000056029b7137a9 in rib_process_dplane_results (thread=<optimized out>) at ../zebra/zebra_rib.c:4810 #12 0x00007fccd75a0e0d in thread_call (thread=thread@entry=0x7ffe8e553cc0) at ../lib/thread.c:1990 #13 0x00007fccd7559368 in frr_run (master=0x56029d65a040) at ../lib/libfrr.c:1198 sonic-net#14 0x000056029b6ac317 in main (argc=9, argv=0x7ffe8e5540d8) at ../zebra/main.c:478

fastiuk added 12 commits February 4, 2024 04:03

System-ready: Update YANG model

7440088

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

System-ready: Add YANG model tests

452fb0c

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

System-ready: Update configuration reference

3f1e0c4

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

System-ready: Extend sysmonitor to listen for daemons

dee1dc2

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

System-ready: Report syncd ready status

937623e

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

System-ready: Update features' configuration

06e977a

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

System-ready: SNMP should wait for system ready event

37b97c8

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

System-ready: Add sysmonitor tests

9508de9

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

🐞 sysmonitor: Skip error logging of inactive srvs

44c4bf0

Signed-off-by: Yevhen Fastiuk <yfastiuk@nvidia.com>

Fix tests

e7a5d3a

Fixup services to enable

f4006dd

Fixup sysmonitor

cac3ed8

fastiuk commented Feb 7, 2024

View reviewed changes

dgsudharsan suggested changes Feb 14, 2024

View reviewed changes

fastiuk closed this Apr 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dev sysready extension#13

Dev sysready extension#13
fastiuk wants to merge 12 commits intomaster-sysready-extensionfrom
dev-sysready-extension

fastiuk commented Feb 5, 2024

Uh oh!

fastiuk Feb 7, 2024

Uh oh!

dgsudharsan Feb 14, 2024

Uh oh!

fastiuk Feb 7, 2024

Uh oh!

fastiuk Feb 7, 2024

Uh oh!

dgsudharsan Feb 14, 2024

Uh oh!

dgsudharsan Feb 14, 2024

Uh oh!

dgsudharsan Feb 14, 2024

Uh oh!

Uh oh!

dgsudharsan Feb 14, 2024

Uh oh!

dgsudharsan Feb 14, 2024

Uh oh!

dgsudharsan Feb 14, 2024

Uh oh!

dgsudharsan Feb 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

fastiuk commented Feb 5, 2024

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants