ci: try to make git work in github actions #2

rjarry · 2024-03-21T09:14:53Z

No description provided.

Signed-off-by: Robin Jarry <[email protected]>

Fix the following crash when running `show ip6 router-advert`: ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f4fb3aba5b8 at pc 0x000000518eeb bp 0x7fff0de9b2b0 sp 0x7fff0de9b2a8 READ of size 8 at 0x7f4fb3aba5b8 thread T0 #0 0x518eea in gr_vec_len ../main/gr_vec.h:35 DPDK#1 0x5197f5 in iface_ra_show ../modules/ip6/control/router_advert.c:90 DPDK#2 0x4050e3 in read_cb ../main/api.c:287 ... DPDK#5 0x409566 in main ../main/main.c:210 The hoplist structure contains a gr_vec of nexthop objects. Fixes: 8ed8b9d ("ip6: router advertisement configuration") Signed-off-by: Robin Jarry <[email protected]>

grout# show ip6 router-advert error: command failed: Bad message grout# ^D ==174902==ERROR: LeakSanitizer: detected memory leaks Direct leak of 328 byte(s) in 1 object(s) allocated from: #0 0x7fe0ca6c2250 in calloc (/lib64/libasan.so.8+0xc2250) DPDK#1 0x7fe0cadef910 in scols_new_table (/lib64/libsmartcols.so.1+0x6910) DPDK#2 0x417d0f in ra_show ../modules/ip6/cli/router_advert.c:19 DPDK#3 0x41e836 in exec_strvec ../cli/exec.c:72 DPDK#4 0x41e982 in exec_line ../cli/exec.c:106 DPDK#5 0x41f412 in interact ../cli/interact.c:119 DPDK#6 0x4216db in main ../cli/main.c:167 ... DPDK#9 0x4017f4 in _start (/root/grout/build/grcli+0x4017f4) Only allocate the table if the GR_IP6_IFACE_RA_SHOW api call is successful. Fixes: 8ed8b9d ("ip6: router advertisement configuration") Signed-off-by: Robin Jarry <[email protected]>

Fix the following crash when running `show ip6 router-advert`: ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f4fb3aba5b8 at pc 0x000000518eeb bp 0x7fff0de9b2b0 sp 0x7fff0de9b2a8 READ of size 8 at 0x7f4fb3aba5b8 thread T0 #0 0x518eea in gr_vec_len ../main/gr_vec.h:35 #1 0x5197f5 in iface_ra_show ../modules/ip6/control/router_advert.c:90 #2 0x4050e3 in read_cb ../main/api.c:287 ... #5 0x409566 in main ../main/main.c:210 The hoplist structure contains a gr_vec of nexthop objects. Fixes: 8ed8b9d ("ip6: router advertisement configuration") Signed-off-by: Robin Jarry <[email protected]>

grout# show ip6 router-advert error: command failed: Bad message grout# ^D ==174902==ERROR: LeakSanitizer: detected memory leaks Direct leak of 328 byte(s) in 1 object(s) allocated from: #0 0x7fe0ca6c2250 in calloc (/lib64/libasan.so.8+0xc2250) #1 0x7fe0cadef910 in scols_new_table (/lib64/libsmartcols.so.1+0x6910) #2 0x417d0f in ra_show ../modules/ip6/cli/router_advert.c:19 #3 0x41e836 in exec_strvec ../cli/exec.c:72 #4 0x41e982 in exec_line ../cli/exec.c:106 #5 0x41f412 in interact ../cli/interact.c:119 #6 0x4216db in main ../cli/main.c:167 ... #9 0x4017f4 in _start (/root/grout/build/grcli+0x4017f4) Only allocate the table if the GR_IP6_IFACE_RA_SHOW api call is successful. Fixes: 8ed8b9d ("ip6: router advertisement configuration") Signed-off-by: Robin Jarry <[email protected]>

When changing any interface type specific configuration, a NULL pointer dereferences happens. Example when changing the number of RXQs of a port: grout> set interface port p0 rxqs 2 ==566931==ERROR: AddressSanitizer: SEGV on unknown address ==566931==The signal is caused by a READ memory access. ==566931==Hint: address points to the zero page. #0 0x455ad5 in iface_port_reconfig ../modules/infra/control/port.c:166 DPDK#1 0x42fbcd in iface_reconfig ../modules/infra/control/iface.c:136 DPDK#2 0x40ee3e in iface_set ../modules/infra/api/iface.c:109 DPDK#3 0x406549 in read_cb ../main/api.c:287 Thread 1 "grout" hit Breakpoint 2, iface_port_reconfig (iface=0x7ff7f39f7f00, set_attrs=4294967296, conf=0x513000008e00, api_info=0x0) at ../modules/infra/control/port.c:154 154 struct iface_info_port *p = (struct iface_info_port *)iface->info; (gdb) p api_info $1 = (const void *) 0x0 The same error may happen when changing the configuration of a vlan or ipip interface. Make sure to pass the type specific information down to iface_reconfig as it is done for iface_create. Update the config smoke test to ensure it works. Fixes: d221e11 ("infra: rework iface creation reconfig apis") Signed-off-by: Robin Jarry <[email protected]>

When changing any interface specific configuration (port rxqs, vlan number, etc.) a NULL pointer dereferences happens. Example when changing the number of RXQs of a port: grout> set interface port p0 rxqs 2 ==566931==ERROR: AddressSanitizer: SEGV on unknown address ==566931==The signal is caused by a READ memory access. ==566931==Hint: address points to the zero page. #0 0x455ad5 in iface_port_reconfig ../modules/infra/control/port.c:166 DPDK#1 0x42fbcd in iface_reconfig ../modules/infra/control/iface.c:136 DPDK#2 0x40ee3e in iface_set ../modules/infra/api/iface.c:109 DPDK#3 0x406549 in read_cb ../main/api.c:287 Thread 1 "grout" hit Breakpoint 2, iface_port_reconfig (iface=0x7ff7f39f7f00, set_attrs=4294967296, conf=0x513000008e00, api_info=0x0) at ../modules/infra/control/port.c:154 154 struct iface_info_port *p = (struct iface_info_port *)iface->info; (gdb) p api_info $1 = (const void *) 0x0 Make sure to pass the type specific information down to iface_reconfig as it is done for iface_create. Update the config smoke test to ensure it works. Fixes: d221e11 ("infra: rework iface creation reconfig apis") Signed-off-by: Robin Jarry <[email protected]>

When changing any interface type specific configuration, a NULL pointer dereferences happens. Example when changing the number of RXQs of a port: grout> set interface port p0 rxqs 2 ==566931==ERROR: AddressSanitizer: SEGV on unknown address ==566931==The signal is caused by a READ memory access. ==566931==Hint: address points to the zero page. #0 0x455ad5 in iface_port_reconfig ../modules/infra/control/port.c:166 #1 0x42fbcd in iface_reconfig ../modules/infra/control/iface.c:136 #2 0x40ee3e in iface_set ../modules/infra/api/iface.c:109 #3 0x406549 in read_cb ../main/api.c:287 Thread 1 "grout" hit Breakpoint 2, iface_port_reconfig (iface=0x7ff7f39f7f00, set_attrs=4294967296, conf=0x513000008e00, api_info=0x0) at ../modules/infra/control/port.c:154 154 struct iface_info_port *p = (struct iface_info_port *)iface->info; (gdb) p api_info $1 = (const void *) 0x0 The same error may happen when changing the configuration of a vlan or ipip interface. Make sure to pass the type specific information down to iface_reconfig as it is done for iface_create. Update the config smoke test to ensure it works. Fixes: d221e11 ("infra: rework iface creation reconfig apis") Signed-off-by: Robin Jarry <[email protected]>

During shutdown, the iface module destroys all interfaces. The IPv4 and IPv6 address modules have event handlers for GR_EVENT_IFACE_PRE_REMOVE which call rib4_cleanup and rib6_cleanup respectively. Currently, there is nothing that guarantees that either the rib nor fib modules are still "active" at the moment these cleanup functions are called. If these modules have already been "finished", this leads to a NULL pointer dereference on shutdown: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004b1a58 in get_rib (vrf_id=0) at ../modules/ip/control/route.c:42 42 rib = vrf_ribs[vrf_id]; [Current thread is 1 (Thread 0x7fb64ab0d900 (LWP 106906))] (gdb) bt #0 0x00000000004b1a58 in get_rib (vrf_id=0) at ../modules/ip/control/route.c:42 DPDK#1 rib4_delete (vrf_id=0, ip=ip@entry=4261417132, prefixlen=prefixlen@entry=24 '\030') at ../modules/ip/control/route.c:178 DPDK#2 0x00000000004b1be3 in rib4_cleanup (nh=0x174971c80) at ../modules/ip/control/route.c:409 DPDK#3 0x00000000004af95c in iface_pre_remove_cb (obj=<optimized out>) at ../modules/ip/control/address.c:185 DPDK#4 0x00000000004a0c8d in gr_event_push (ev_type=ev_type@entry=2900099074, obj=obj@entry=0x1739f6280) at ../main/event.c:24 DPDK#5 0x00000000004a4707 in iface_destroy (ifid=ifid@entry=1) at ../modules/infra/control/iface.c:253 DPDK#6 0x00000000004a48b6 in iface_fini () at ../modules/infra/control/iface.c:295 DPDK#7 0x0000000000403182 in modules_fini (ev_base=ev_base@entry=0x17afeed0) at ../main/module.c:178 DPDK#8 0x0000000000402f1a in main (argc=<optimized out>, argv=<optimized out>) at ../main/main.c:303 Add a dependency to both ipv4 and ipv6 route modules so that they are still accessible when the iface module is finalized. Signed-off-by: Robin Jarry <[email protected]>

During shutdown, the iface module destroys all interfaces. The IPv4 and IPv6 address modules have event handlers for GR_EVENT_IFACE_PRE_REMOVE which call rib4_cleanup and rib6_cleanup respectively. Currently, there is nothing that guarantees that either the rib nor fib modules are still "active" at the moment these cleanup functions are called. If these modules have already been "finished", this leads to a NULL pointer dereference on shutdown: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004b1a58 in get_rib (vrf_id=0) at ../modules/ip/control/route.c:42 42 rib = vrf_ribs[vrf_id]; [Current thread is 1 (Thread 0x7fb64ab0d900 (LWP 106906))] (gdb) bt #0 0x00000000004b1a58 in get_rib (vrf_id=0) at ../modules/ip/control/route.c:42 #1 rib4_delete (vrf_id=0, ip=ip@entry=4261417132, prefixlen=prefixlen@entry=24 '\030') at ../modules/ip/control/route.c:178 #2 0x00000000004b1be3 in rib4_cleanup (nh=0x174971c80) at ../modules/ip/control/route.c:409 #3 0x00000000004af95c in iface_pre_remove_cb (obj=<optimized out>) at ../modules/ip/control/address.c:185 #4 0x00000000004a0c8d in gr_event_push (ev_type=ev_type@entry=2900099074, obj=obj@entry=0x1739f6280) at ../main/event.c:24 #5 0x00000000004a4707 in iface_destroy (ifid=ifid@entry=1) at ../modules/infra/control/iface.c:253 #6 0x00000000004a48b6 in iface_fini () at ../modules/infra/control/iface.c:295 #7 0x0000000000403182 in modules_fini (ev_base=ev_base@entry=0x17afeed0) at ../main/module.c:178 #8 0x0000000000402f1a in main (argc=<optimized out>, argv=<optimized out>) at ../main/main.c:303 Add a dependency to both ipv4 and ipv6 route modules so that they are still accessible when the iface module is finalized. Signed-off-by: Robin Jarry <[email protected]>

When adding an address to an interface, we use gr_vec_add() which may call realloc() internally and may free the old memory area. This can cause use after free bugs as detected by libasan in the CI: + grcli interface add port gm2dgn1 devargs net_tap1,iface=gm2dgn1 mac f0:0d:ac:dc:00:01 ... DEBUG: GROUT: iface_event: iface event [0xacdc0001] POST_ADD triggered for iface gm2dgn1. INFO: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff02::1:ffdc:1 INFO: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff01::1 INFO: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff02::1 INFO: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff01::2 NOTICE: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff02::2 GROUT: trace_log_packet: [rx gm2dgn1] f0:0d:ac:dc:00:01 > 33:33:00:00:00:16 / IPv6 :: > ff02::16 ttl=1 proto=HOPOPT(0) / ICMPv6 type=143 code=0, (pkt_len=90) ================================================================= ==90746==ERROR: AddressSanitizer: heap-use-after-free on address 0x504000006390 at pc 0x560cd400d1b7 bp 0x7f6cbdfd7760 sp 0x7f6cbdfd7750 READ of size 8 at 0x504000006390 thread T4 #0 0x560cd400d1b6 in __gr_vec_hdr ../main/gr_vec.h:34 DPDK#1 0x560cd400d284 in gr_vec_len ../main/gr_vec.h:50 DPDK#2 0x560cd400dc98 in mcast6_get_member ../modules/ip6/control/address.c:92 DPDK#3 0x560cd404d30e in ip6_input_process ../modules/ip6/datapath/ip6_input.c:96 DPDK#4 0x560cd3f8ef05 in __rte_node_process ../subprojects/dpdk/lib/graph/rte_graph_worker_common.h:207 DPDK#5 0x560cd3f8ef05 in rte_graph_walk_rtc ../subprojects/dpdk/lib/graph/rte_graph_model_rtc.h:42 DPDK#6 0x560cd3f8f78f in rte_graph_walk ../subprojects/dpdk/lib/graph/rte_graph_worker.h:38 DPDK#7 0x560cd3f91c1c in gr_datapath_loop ../modules/infra/datapath/main_loop.c:252 ... 0x504000006390 is located 0 bytes inside of 48-byte region [0x504000006390,0x5040000063c0) freed by thread T0 here: #0 0x7f6d498fc778 in realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:85 DPDK#1 0x560cd400d364 in __gr_vec_grow ../main/gr_vec.h:69 DPDK#2 0x560cd400e68c in mcast6_addr_add ../modules/ip6/control/address.c:141 DPDK#3 0x560cd4010bc6 in ip6_iface_event_handler ../modules/ip6/control/address.c:373 DPDK#4 0x560cd3f0209e in gr_event_push ../main/event.c:24 DPDK#5 0x560cd3f2d9b0 in iface_create ../modules/infra/control/iface.c:134 DPDK#6 0x560cd3f08c1a in iface_add ../modules/infra/api/iface.c:33 ... Prevent this from happening by cloning the address vector before adding anything to it. Add the new address to the clone and make it visible to datapath threads. Use the RCU to ensure all threads have seen the new vector and *then* free the old vector. Link: https://github.com/DPDK/grout/actions/runs/18442009960/job/52543656749#step:9:4363 Signed-off-by: Robin Jarry <[email protected]>

Fix the following crash when running `show ip6 router-advert`: ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f4fb3aba5b8 at pc 0x000000518eeb bp 0x7fff0de9b2b0 sp 0x7fff0de9b2a8 READ of size 8 at 0x7f4fb3aba5b8 thread T0 #0 0x518eea in gr_vec_len ../main/gr_vec.h:35 DPDK#1 0x5197f5 in iface_ra_show ../modules/ip6/control/router_advert.c:90 DPDK#2 0x4050e3 in read_cb ../main/api.c:287 ... DPDK#5 0x409566 in main ../main/main.c:210 The hoplist structure contains a gr_vec of nexthop objects. Fixes: 8ed8b9d ("ip6: router advertisement configuration") Signed-off-by: Robin Jarry <[email protected]> Reviewed-by: Christophe Fontaine <[email protected]>

grout# show ip6 router-advert error: command failed: Bad message grout# ^D ==174902==ERROR: LeakSanitizer: detected memory leaks Direct leak of 328 byte(s) in 1 object(s) allocated from: #0 0x7fe0ca6c2250 in calloc (/lib64/libasan.so.8+0xc2250) DPDK#1 0x7fe0cadef910 in scols_new_table (/lib64/libsmartcols.so.1+0x6910) DPDK#2 0x417d0f in ra_show ../modules/ip6/cli/router_advert.c:19 DPDK#3 0x41e836 in exec_strvec ../cli/exec.c:72 DPDK#4 0x41e982 in exec_line ../cli/exec.c:106 DPDK#5 0x41f412 in interact ../cli/interact.c:119 DPDK#6 0x4216db in main ../cli/main.c:167 ... DPDK#9 0x4017f4 in _start (/root/grout/build/grcli+0x4017f4) Only allocate the table if the GR_IP6_IFACE_RA_SHOW api call is successful. Fixes: 8ed8b9d ("ip6: router advertisement configuration") Signed-off-by: Robin Jarry <[email protected]> Reviewed-by: Christophe Fontaine <[email protected]>

When changing any interface type specific configuration, a NULL pointer dereferences happens. Example when changing the number of RXQs of a port: grout> set interface port p0 rxqs 2 ==566931==ERROR: AddressSanitizer: SEGV on unknown address ==566931==The signal is caused by a READ memory access. ==566931==Hint: address points to the zero page. #0 0x455ad5 in iface_port_reconfig ../modules/infra/control/port.c:166 DPDK#1 0x42fbcd in iface_reconfig ../modules/infra/control/iface.c:136 DPDK#2 0x40ee3e in iface_set ../modules/infra/api/iface.c:109 DPDK#3 0x406549 in read_cb ../main/api.c:287 Thread 1 "grout" hit Breakpoint 2, iface_port_reconfig (iface=0x7ff7f39f7f00, set_attrs=4294967296, conf=0x513000008e00, api_info=0x0) at ../modules/infra/control/port.c:154 154 struct iface_info_port *p = (struct iface_info_port *)iface->info; (gdb) p api_info $1 = (const void *) 0x0 The same error may happen when changing the configuration of a vlan or ipip interface. Make sure to pass the type specific information down to iface_reconfig as it is done for iface_create. Update the config smoke test to ensure it works. Fixes: d221e11 ("infra: rework iface creation reconfig apis") Signed-off-by: Robin Jarry <[email protected]> Reviewed-by: Christophe Fontaine <[email protected]>

During shutdown, the iface module destroys all interfaces. The IPv4 and IPv6 address modules have event handlers for GR_EVENT_IFACE_PRE_REMOVE which call rib4_cleanup and rib6_cleanup respectively. Currently, there is nothing that guarantees that either the rib nor fib modules are still "active" at the moment these cleanup functions are called. If these modules have already been "finished", this leads to a NULL pointer dereference on shutdown: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004b1a58 in get_rib (vrf_id=0) at ../modules/ip/control/route.c:42 42 rib = vrf_ribs[vrf_id]; [Current thread is 1 (Thread 0x7fb64ab0d900 (LWP 106906))] (gdb) bt #0 0x00000000004b1a58 in get_rib (vrf_id=0) at ../modules/ip/control/route.c:42 DPDK#1 rib4_delete (vrf_id=0, ip=ip@entry=4261417132, prefixlen=prefixlen@entry=24 '\030') at ../modules/ip/control/route.c:178 DPDK#2 0x00000000004b1be3 in rib4_cleanup (nh=0x174971c80) at ../modules/ip/control/route.c:409 DPDK#3 0x00000000004af95c in iface_pre_remove_cb (obj=<optimized out>) at ../modules/ip/control/address.c:185 DPDK#4 0x00000000004a0c8d in gr_event_push (ev_type=ev_type@entry=2900099074, obj=obj@entry=0x1739f6280) at ../main/event.c:24 DPDK#5 0x00000000004a4707 in iface_destroy (ifid=ifid@entry=1) at ../modules/infra/control/iface.c:253 DPDK#6 0x00000000004a48b6 in iface_fini () at ../modules/infra/control/iface.c:295 DPDK#7 0x0000000000403182 in modules_fini (ev_base=ev_base@entry=0x17afeed0) at ../main/module.c:178 DPDK#8 0x0000000000402f1a in main (argc=<optimized out>, argv=<optimized out>) at ../main/main.c:303 Add a dependency to both ipv4 and ipv6 route modules so that they are still accessible when the iface module is finalized. Signed-off-by: Robin Jarry <[email protected]> Reviewed-by: Christophe Fontaine <[email protected]>

When adding an address to an interface, we use gr_vec_add() which may call realloc() internally and may free the old memory area. This can cause use after free bugs as detected by libasan in the CI: + grcli interface add port gm2dgn1 devargs net_tap1,iface=gm2dgn1 mac f0:0d:ac:dc:00:01 ... DEBUG: GROUT: iface_event: iface event [0xacdc0001] POST_ADD triggered for iface gm2dgn1. INFO: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff02::1:ffdc:1 INFO: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff01::1 INFO: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff02::1 INFO: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff01::2 NOTICE: GROUT: mcast6_addr_add: gm2dgn1: joining multicast group ff02::2 GROUT: trace_log_packet: [rx gm2dgn1] f0:0d:ac:dc:00:01 > 33:33:00:00:00:16 / IPv6 :: > ff02::16 ttl=1 proto=HOPOPT(0) / ICMPv6 type=143 code=0, (pkt_len=90) ================================================================= ==90746==ERROR: AddressSanitizer: heap-use-after-free on address 0x504000006390 at pc 0x560cd400d1b7 bp 0x7f6cbdfd7760 sp 0x7f6cbdfd7750 READ of size 8 at 0x504000006390 thread T4 #0 0x560cd400d1b6 in __gr_vec_hdr ../main/gr_vec.h:34 DPDK#1 0x560cd400d284 in gr_vec_len ../main/gr_vec.h:50 DPDK#2 0x560cd400dc98 in mcast6_get_member ../modules/ip6/control/address.c:92 DPDK#3 0x560cd404d30e in ip6_input_process ../modules/ip6/datapath/ip6_input.c:96 DPDK#4 0x560cd3f8ef05 in __rte_node_process ../subprojects/dpdk/lib/graph/rte_graph_worker_common.h:207 DPDK#5 0x560cd3f8ef05 in rte_graph_walk_rtc ../subprojects/dpdk/lib/graph/rte_graph_model_rtc.h:42 DPDK#6 0x560cd3f8f78f in rte_graph_walk ../subprojects/dpdk/lib/graph/rte_graph_worker.h:38 DPDK#7 0x560cd3f91c1c in gr_datapath_loop ../modules/infra/datapath/main_loop.c:252 ... 0x504000006390 is located 0 bytes inside of 48-byte region [0x504000006390,0x5040000063c0) freed by thread T0 here: #0 0x7f6d498fc778 in realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:85 DPDK#1 0x560cd400d364 in __gr_vec_grow ../main/gr_vec.h:69 DPDK#2 0x560cd400e68c in mcast6_addr_add ../modules/ip6/control/address.c:141 DPDK#3 0x560cd4010bc6 in ip6_iface_event_handler ../modules/ip6/control/address.c:373 DPDK#4 0x560cd3f0209e in gr_event_push ../main/event.c:24 DPDK#5 0x560cd3f2d9b0 in iface_create ../modules/infra/control/iface.c:134 DPDK#6 0x560cd3f08c1a in iface_add ../modules/infra/api/iface.c:33 ... Prevent this from happening by cloning the address vector before adding anything to it. Add the new address to the clone and make it visible to datapath threads. Use the RCU to ensure all threads have seen the new vector and *then* free the old vector. Link: https://github.com/DPDK/grout/actions/runs/18442009960/job/52543656749#step:9:4363 Signed-off-by: Robin Jarry <[email protected]> Reviewed-by: Christophe Fontaine <[email protected]>

When an interface or nexthop is deleted while packets referencing them are still queued in the control output ring, callbacks would be invoked with dangling pointers, causing heap-use-after-free errors. Example backtrace from smoke tests: ==63400==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f1a06511100 READ of size 2 at 0x7f1a06511100 thread T0 #0 in ndp_router_sollicit_input_cb ../modules/ip6/control/router_advert.c:109 DPDK#1 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#2 in event_base_loop DPDK#3 in main ../main/main.c:314 Introduce a handler registration mechanism where each control output callback can provide a cleanup check function. When interfaces or nexthops are deleted, the control output ring is immediately drained. During the drain, packets are checked against registered handlers. If a handler's cleanup function indicates the packet references the deleted object, the packet is freed without invoking the callback. Both GR_EVENT_IFACE_REMOVE and GR_EVENT_NEXTHOP_DELETE events are moved to occur after RCU synchronization. This ensures all datapath threads have seen that the objects are no longer accessible (removed from their respective lookup tables before the sync). At that point, no new packets referencing the deleted objects can be enqueued and only packets already in the control output ring may still hold references. The events trigger a cleanup drain before the object memory is freed, preventing callbacks from accessing freed memory. Signed-off-by: Robin Jarry <[email protected]>

When an interface or nexthop is deleted while packets referencing them are still queued in the control output ring, callbacks would be invoked with dangling pointers, causing heap-use-after-free errors. Example backtrace from smoke tests: ERROR: AddressSanitizer: heap-use-after-free on address 0x7f1a06511100 READ of size 2 at 0x7f1a06511100 thread T0 #0 in ndp_router_sollicit_input_cb ../modules/ip6/control/router_advert.c:109 DPDK#1 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#2 in event_base_loop DPDK#3 in main ../main/main.c:314 Introduce a handler registration mechanism where each control output callback can provide a cleanup check function. When interfaces or nexthops are deleted, the control output ring is immediately drained. During the drain, packets are checked against registered handlers. If a handler's cleanup function indicates the packet references the deleted object, the packet is freed without invoking the callback. Both GR_EVENT_IFACE_REMOVE and GR_EVENT_NEXTHOP_DELETE events are moved to occur after RCU synchronization. This ensures all datapath threads have seen that the objects are no longer accessible (removed from their respective lookup tables before the sync). At that point, no new packets referencing the deleted objects can be enqueued and only packets already in the control output ring may still hold references. The events trigger a cleanup drain before the object memory is freed, preventing callbacks from accessing freed memory. Signed-off-by: Robin Jarry <[email protected]> Co-authored-by: Christophe Fontaine <[email protected]>

When an interface or nexthop is deleted while packets referencing them are still queued in the control output ring, callbacks would be invoked with dangling pointers, causing heap-use-after-free errors. Example backtrace from smoke tests: ==63400==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f1a06511100 READ of size 2 at 0x7f1a06511100 thread T0 #0 in ndp_router_sollicit_input_cb ../modules/ip6/control/router_advert.c:109 DPDK#1 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#2 in event_base_loop DPDK#3 in main ../main/main.c:314 Add a drain parameter to all control_output_cb_t callbacks. When interfaces or nexthops are deleted, the control output ring is drained with a control_output_drain context containing the event type and deleted object pointer. Callbacks check this context and free packets that reference the deleted object without processing them. Both GR_EVENT_IFACE_REMOVE and GR_EVENT_NEXTHOP_DELETE events are moved to occur after RCU synchronization. This ensures all datapath threads have seen that the objects are no longer accessible (removed from their respective lookup tables before the sync). At that point, no new packets referencing the deleted objects can be enqueued and only packets already in the control output ring may still hold references. The events trigger a drain before the object memory is freed, preventing callbacks from accessing freed memory. Suggested-by: Maxime Leroy <[email protected]> Signed-off-by: Robin Jarry <[email protected]> Co-authored-by: Christophe Fontaine <[email protected]>

When an interface or nexthop is deleted while packets referencing them are still queued in the control output ring, callbacks would be invoked with dangling pointers, causing heap-use-after-free errors. Example backtrace from smoke tests: ERROR: AddressSanitizer: heap-use-after-free on address 0x7f1a06511100 READ of size 2 at 0x7f1a06511100 thread T0 #0 in ndp_router_sollicit_input_cb ../modules/ip6/control/router_advert.c:109 DPDK#1 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#2 in event_base_loop DPDK#3 in main ../main/main.c:314 Add a drain parameter to all control_output_cb_t callbacks. When interfaces or nexthops are deleted, the control output ring is drained with a control_output_drain context containing the event type and deleted object pointer. Callbacks check this context and free packets that reference the deleted object without processing them. Both GR_EVENT_IFACE_REMOVE and GR_EVENT_NEXTHOP_DELETE events are moved to occur after RCU synchronization. This ensures all datapath threads have seen that the objects are no longer accessible (removed from their respective lookup tables before the sync). At that point, no new packets referencing the deleted objects can be enqueued and only packets already in the control output ring may still hold references. The events trigger a drain before the object memory is freed, preventing callbacks from accessing freed memory. Suggested-by: Maxime Leroy <[email protected]> Signed-off-by: Robin Jarry <[email protected]> Co-authored-by: Christophe Fontaine <[email protected]>

When an interface or nexthop is deleted while packets referencing them are still queued in the control output ring, callbacks would be invoked with dangling pointers, causing heap-use-after-free errors. Example backtrace from smoke tests: ERROR: AddressSanitizer: heap-use-after-free on address 0x7f1a06511100 READ of size 2 at 0x7f1a06511100 thread T0 #0 in ndp_router_sollicit_input_cb ../modules/ip6/control/router_advert.c:109 DPDK#1 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#2 in event_base_loop DPDK#3 in main ../main/main.c:314 Add a drain parameter to all control_output_cb_t callbacks. When interfaces or nexthops are deleted, the control output ring is drained with a control_output_drain context containing the event type and deleted object pointer. Callbacks check this context and free packets that reference the deleted object without processing them. Both GR_EVENT_IFACE_REMOVE and GR_EVENT_NEXTHOP_DELETE events are moved to occur after RCU synchronization. This ensures all datapath threads have seen that the objects are no longer accessible (removed from their respective lookup tables before the sync). At that point, no new packets referencing the deleted objects can be enqueued and only packets already in the control output ring may still hold references. The events trigger a drain before the object memory is freed, preventing callbacks from accessing freed memory. Suggested-by: Maxime Leroy <[email protected]> Signed-off-by: Robin Jarry <[email protected]> Co-authored-by: Christophe Fontaine <[email protected]> Reviewed-by: Maxime Leroy <[email protected]>

When an interface or nexthop is deleted while packets referencing them are still queued in the control output ring, callbacks would be invoked with dangling pointers, causing heap-use-after-free errors. Example backtrace from smoke tests: ERROR: AddressSanitizer: heap-use-after-free on address 0x7f1a06511100 READ of size 2 at 0x7f1a06511100 thread T0 #0 in ndp_router_sollicit_input_cb ../modules/ip6/control/router_advert.c:109 #1 in control_output_poll ../modules/infra/control/control_output.c:33 #2 in event_base_loop #3 in main ../main/main.c:314 Add a drain parameter to all control_output_cb_t callbacks. When interfaces or nexthops are deleted, the control output ring is drained with a control_output_drain context containing the event type and deleted object pointer. Callbacks check this context and free packets that reference the deleted object without processing them. Both GR_EVENT_IFACE_REMOVE and GR_EVENT_NEXTHOP_DELETE events are moved to occur after RCU synchronization. This ensures all datapath threads have seen that the objects are no longer accessible (removed from their respective lookup tables before the sync). At that point, no new packets referencing the deleted objects can be enqueued and only packets already in the control output ring may still hold references. The events trigger a drain before the object memory is freed, preventing callbacks from accessing freed memory. Suggested-by: Maxime Leroy <[email protected]> Signed-off-by: Robin Jarry <[email protected]> Co-authored-by: Christophe Fontaine <[email protected]> Reviewed-by: Maxime Leroy <[email protected]>

Since commit e954453 ("control-output: prevent use-after-free on object deletion"), deleting interfaces or nexthops causes a call to control_output_poll() in order to drain the packets that may reference the deleted object. On shutdown, the modules finalization order causes control_output to be finalized before the iface module. When the iface module is finalized, all remaining interfaces are deleted and control_output_poll() accesses the ring that was already "freed". ERROR: AddressSanitizer: heap-use-after-free on address 0x00016d19d608 at pc 0x00000045901c bp 0xffffca318bd0 sp 0xffffca318be8 READ of size 4 at 0x00016d19d608 thread T0 #0 0x000000459018 in rte_ring_dequeue_bulk_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:375 DPDK#1 0x000000459018 in rte_ring_dequeue_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:471 DPDK#2 0x000000459018 in rte_ring_dequeue ../subprojects/dpdk/lib/ring/rte_ring.h:496 DPDK#3 0x000000459018 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#4 0x0000004648a0 in event_handler ../modules/infra/control/control_output.c:76 DPDK#5 0x000000416974 in gr_event_push ../main/event.c:26 DPDK#6 0x00000046a930 in iface_destroy ../modules/infra/control/iface.c:424 DPDK#7 0x00000046b138 in iface_fini ../modules/infra/control/iface.c:462 DPDK#8 0x00000041d63c in modules_fini ../main/module.c:112 DPDK#9 0x00000041b5a4 in main ../main/main.c:326 Ensure both iface and nexthop are finalized before control_output. Fixes: e954453 ("control-output: prevent use-after-free on object deletion") Signed-off-by: Robin Jarry <[email protected]>

Since commit e954453 ("control-output: prevent use-after-free on object deletion"), deleting interfaces or nexthops causes a call to control_output_poll() in order to drain the packets that may reference the deleted object. On shutdown, the modules finalization order causes control_output to be finalized before the iface module. When the iface module is finalized, all remaining interfaces are deleted and control_output_poll() accesses the ring that was already "freed". ERROR: AddressSanitizer: heap-use-after-free on address 0x00016d19d608 at pc 0x00000045901c bp 0xffffca318bd0 sp 0xffffca318be8 READ of size 4 at 0x00016d19d608 thread T0 #0 0x000000459018 in rte_ring_dequeue_bulk_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:375 DPDK#1 0x000000459018 in rte_ring_dequeue_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:471 DPDK#2 0x000000459018 in rte_ring_dequeue ../subprojects/dpdk/lib/ring/rte_ring.h:496 DPDK#3 0x000000459018 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#4 0x0000004648a0 in event_handler ../modules/infra/control/control_output.c:76 DPDK#5 0x000000416974 in gr_event_push ../main/event.c:26 DPDK#6 0x00000046a930 in iface_destroy ../modules/infra/control/iface.c:424 DPDK#7 0x00000046b138 in iface_fini ../modules/infra/control/iface.c:462 DPDK#8 0x00000041d63c in modules_fini ../main/module.c:112 DPDK#9 0x00000041b5a4 in main ../main/main.c:326 Ensure both iface and nexthop are finalized before control_output. Fixes: e954453 ("control-output: prevent use-after-free on object deletion") Signed-off-by: Robin Jarry <[email protected]> Tested-by: Christophe Fontaine <[email protected]>

Since commit e954453 ("control-output: prevent use-after-free on object deletion"), deleting interfaces or nexthops causes a call to control_output_poll() in order to drain the packets that may reference the deleted object. On shutdown, the modules finalization order causes control_output to be finalized before the iface module. When the iface module is finalized, all remaining interfaces are deleted and control_output_poll() accesses the ring that was already "freed". ERROR: AddressSanitizer: heap-use-after-free on address 0x00016d19d608 at pc 0x00000045901c bp 0xffffca318bd0 sp 0xffffca318be8 READ of size 4 at 0x00016d19d608 thread T0 #0 0x000000459018 in rte_ring_dequeue_bulk_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:375 DPDK#1 0x000000459018 in rte_ring_dequeue_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:471 DPDK#2 0x000000459018 in rte_ring_dequeue ../subprojects/dpdk/lib/ring/rte_ring.h:496 DPDK#3 0x000000459018 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#4 0x0000004648a0 in event_handler ../modules/infra/control/control_output.c:76 DPDK#5 0x000000416974 in gr_event_push ../main/event.c:26 DPDK#6 0x00000046a930 in iface_destroy ../modules/infra/control/iface.c:424 DPDK#7 0x00000046b138 in iface_fini ../modules/infra/control/iface.c:462 DPDK#8 0x00000041d63c in modules_fini ../main/module.c:112 DPDK#9 0x00000041b5a4 in main ../main/main.c:326 Ensure both iface and nexthop are finalized before control_output. Fixes: e954453 ("control-output: prevent use-after-free on object deletion") Signed-off-by: Robin Jarry <[email protected]> Tested-by: Christophe Fontaine <[email protected]> Reviewed-by: Maxime Leroy <[email protected]>

Since commit e954453 ("control-output: prevent use-after-free on object deletion"), deleting interfaces or nexthops causes a call to control_output_poll() in order to drain the packets that may reference the deleted object. On shutdown, the modules finalization order causes control_output to be finalized before the iface module. When the iface module is finalized, all remaining interfaces are deleted and control_output_poll() accesses the ring that was already "freed". ERROR: AddressSanitizer: heap-use-after-free on address 0x00016d19d608 at pc 0x00000045901c bp 0xffffca318bd0 sp 0xffffca318be8 READ of size 4 at 0x00016d19d608 thread T0 #0 0x000000459018 in rte_ring_dequeue_bulk_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:375 DPDK#1 0x000000459018 in rte_ring_dequeue_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:471 DPDK#2 0x000000459018 in rte_ring_dequeue ../subprojects/dpdk/lib/ring/rte_ring.h:496 DPDK#3 0x000000459018 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#4 0x0000004648a0 in event_handler ../modules/infra/control/control_output.c:76 DPDK#5 0x000000416974 in gr_event_push ../main/event.c:26 DPDK#6 0x00000046a930 in iface_destroy ../modules/infra/control/iface.c:424 DPDK#7 0x00000046b138 in iface_fini ../modules/infra/control/iface.c:462 DPDK#8 0x00000041d63c in modules_fini ../main/module.c:112 DPDK#9 0x00000041b5a4 in main ../main/main.c:326 Ensure both iface and nexthop are finalized before control_output. Fixes: e954453 ("control-output: prevent use-after-free on object deletion") Signed-off-by: Robin Jarry <[email protected]> Tested-by: Christophe Fontaine <[email protected]>

Since commit e954453 ("control-output: prevent use-after-free on object deletion"), deleting interfaces or nexthops causes a call to control_output_poll() in order to drain the packets that may reference the deleted object. On shutdown, the modules finalization order causes control_output to be finalized before the iface module. When the iface module is finalized, all remaining interfaces are deleted and control_output_poll() accesses the ring that was already "freed". ERROR: AddressSanitizer: heap-use-after-free on address 0x00016d19d608 at pc 0x00000045901c bp 0xffffca318bd0 sp 0xffffca318be8 READ of size 4 at 0x00016d19d608 thread T0 #0 0x000000459018 in rte_ring_dequeue_bulk_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:375 DPDK#1 0x000000459018 in rte_ring_dequeue_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:471 DPDK#2 0x000000459018 in rte_ring_dequeue ../subprojects/dpdk/lib/ring/rte_ring.h:496 DPDK#3 0x000000459018 in control_output_poll ../modules/infra/control/control_output.c:33 DPDK#4 0x0000004648a0 in event_handler ../modules/infra/control/control_output.c:76 DPDK#5 0x000000416974 in gr_event_push ../main/event.c:26 DPDK#6 0x00000046a930 in iface_destroy ../modules/infra/control/iface.c:424 DPDK#7 0x00000046b138 in iface_fini ../modules/infra/control/iface.c:462 DPDK#8 0x00000041d63c in modules_fini ../main/module.c:112 DPDK#9 0x00000041b5a4 in main ../main/main.c:326 Ensure both iface and nexthop are finalized before control_output. Fixes: e954453 ("control-output: prevent use-after-free on object deletion") Signed-off-by: Robin Jarry <[email protected]> Tested-by: Christophe Fontaine <[email protected]> Reviewed-by: Maxime Leroy <[email protected]>

Since commit e954453 ("control-output: prevent use-after-free on object deletion"), deleting interfaces or nexthops causes a call to control_output_poll() in order to drain the packets that may reference the deleted object. On shutdown, the modules finalization order causes control_output to be finalized before the iface module. When the iface module is finalized, all remaining interfaces are deleted and control_output_poll() accesses the ring that was already "freed". ERROR: AddressSanitizer: heap-use-after-free on address 0x00016d19d608 at pc 0x00000045901c bp 0xffffca318bd0 sp 0xffffca318be8 READ of size 4 at 0x00016d19d608 thread T0 #0 0x000000459018 in rte_ring_dequeue_bulk_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:375 #1 0x000000459018 in rte_ring_dequeue_elem ../subprojects/dpdk/lib/ring/rte_ring_elem.h:471 #2 0x000000459018 in rte_ring_dequeue ../subprojects/dpdk/lib/ring/rte_ring.h:496 #3 0x000000459018 in control_output_poll ../modules/infra/control/control_output.c:33 #4 0x0000004648a0 in event_handler ../modules/infra/control/control_output.c:76 #5 0x000000416974 in gr_event_push ../main/event.c:26 #6 0x00000046a930 in iface_destroy ../modules/infra/control/iface.c:424 #7 0x00000046b138 in iface_fini ../modules/infra/control/iface.c:462 #8 0x00000041d63c in modules_fini ../main/module.c:112 #9 0x00000041b5a4 in main ../main/main.c:326 Ensure both iface and nexthop are finalized before control_output. Fixes: e954453 ("control-output: prevent use-after-free on object deletion") Signed-off-by: Robin Jarry <[email protected]> Tested-by: Christophe Fontaine <[email protected]> Reviewed-by: Maxime Leroy <[email protected]>

When deleting a port interface which is part of a bond, a dangling pointer is left in the bond members array. This causes crashes when destroying the bond: Program terminated with signal SIGSEGV, Segmentation fault. error reading variable: Cannot access memory at address 0xc Thread 1 (Thread 0x7f2f5cb51bc0 (LWP 657645)): #0 rte_arch_bswap16 at ../subprojects/dpdk/lib/eal/x86/include/rte_byteorder.h:30 DPDK#1 bond_update_active_members at ../modules/infra/control/bond.c:248 DPDK#2 gr_event_push at ../main/event.c:26 DPDK#3 iface_destroy at ../modules/infra/control/iface.c:409 DPDK#4 iface_del at ../modules/infra/api/iface.c:60 Make sure to remove the member when a port is deleted. Update the primary index accordingly. Signed-off-by: Robin Jarry <[email protected]>

When deleting a port interface which is part of a bond, a dangling pointer is left in the bond members array. This causes uses-after-free issues accessing bond members: AddressSanitizer: heap-use-after-free on address 0x7b85fbe15100 READ of size 2 at 0x7b85fbe15100 thread T0 #0 bond_to_api ../modules/infra/control/bond.c:399 DPDK#1 iface_to_api ../modules/infra/api/iface.c:24 DPDK#2 iface_list ../modules/infra/api/iface.c:100 Make sure to remove the member when a port is deleted. Update the primary index accordingly. Signed-off-by: Robin Jarry <[email protected]>

When deleting a port interface which is part of a bond, a dangling pointer is left in the bond members array. This causes uses-after-free issues accessing bond members: AddressSanitizer: heap-use-after-free on address 0x7b85fbe15100 READ of size 2 at 0x7b85fbe15100 thread T0 #0 bond_to_api ../modules/infra/control/bond.c:399 DPDK#1 iface_to_api ../modules/infra/api/iface.c:24 DPDK#2 iface_list ../modules/infra/api/iface.c:100 Make sure to remove the member when a port is deleted. Update the primary index accordingly. Signed-off-by: Robin Jarry <[email protected]> Reviewed-by: Christophe Fontaine <[email protected]>

When deleting a port interface which is part of a bond, a dangling pointer is left in the bond members array. This causes uses-after-free issues accessing bond members: AddressSanitizer: heap-use-after-free on address 0x7b85fbe15100 READ of size 2 at 0x7b85fbe15100 thread T0 #0 bond_to_api ../modules/infra/control/bond.c:399 #1 iface_to_api ../modules/infra/api/iface.c:24 #2 iface_list ../modules/infra/api/iface.c:100 Make sure to remove the member when a port is deleted. Update the primary index accordingly. Signed-off-by: Robin Jarry <[email protected]> Reviewed-by: Christophe Fontaine <[email protected]>

Similar to "control_output: fix use-after-free on shutdown", packets are still in the icmp(6) queue when grout quits, which ends up into a 'use-after-free': ==51938==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f6e3084f080 at pc 0x560293c3dc87 bp 0x7ffe3e9298b0 sp 0x7ffe3e9298a0 READ of size 8 at 0x7f6e3084f080 thread T0 #0 0x560293c3dc86 in rte_pktmbuf_free ../subprojects/dpdk/lib/mbuf/rte_mbuf.h:1522 DPDK#1 0x560293c42a8a in icmp6_queue_pop ../modules/ip6/control/icmp6.c:30 DPDK#2 0x560293c49cb8 in icmp_fini ../modules/ip6/control/icmp6.c:177 DPDK#3 0x560293ac1068 in modules_fini ../main/module.c:123 DPDK#4 0x560293abfb52 in main ../main/main.c:326 DPDK#5 0x7f6e5ca2a1c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e) DPDK#6 0x7f6e5ca2a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e) DPDK#7 0x560293ab5974 in _start (/home/runner/work/grout/grout/build/grout+0x312974) (BuildId: 9f73d7b419f769b2197f675b4a3d768d8259e548) Similar to other modules, don't attempt to free individual elements. Signed-off-by: Christophe Fontaine <[email protected]>

rjarry force-pushed the test-pr branch 5 times, most recently from d80d31a to 456a921 Compare March 21, 2024 09:25

rjarry added 2 commits March 21, 2024 12:06

ci: try to make git work in github actions

60170a4

Signed-off-by: Robin Jarry <[email protected]>

clang: fix for older versions

c46221f

Signed-off-by: Robin Jarry <[email protected]>

rjarry force-pushed the test-pr branch from 306517a to c46221f Compare March 21, 2024 11:06

wip

cd6b94d

Signed-off-by: Robin Jarry <[email protected]>

rjarry force-pushed the test-pr branch from 2b1ebc1 to 7382fee Compare March 21, 2024 11:17

wip

6c24436

Signed-off-by: Robin Jarry <[email protected]>

rjarry force-pushed the test-pr branch from 7382fee to 6c24436 Compare March 21, 2024 11:25

wip

d10c376

Signed-off-by: Robin Jarry <[email protected]>

rjarry force-pushed the test-pr branch from d41a4f2 to d10c376 Compare March 21, 2024 11:31

rjarry closed this Mar 21, 2024

rjarry deleted the test-pr branch March 21, 2024 11:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: try to make git work in github actions #2

ci: try to make git work in github actions #2

Uh oh!

rjarry commented Mar 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ci: try to make git work in github actions #2

ci: try to make git work in github actions #2

Uh oh!

Conversation

rjarry commented Mar 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants