MCLAG enhacements ICCPd initial code commit by Praveen-Brcm · Pull Request #4819 · sonic-net/sonic-buildimage

Praveen-Brcm · 2020-06-19T15:50:28Z

Why I did it
This PR contains the ICCPd changes to support MCLAG enhancements feature.
How I did it
Implemented as per the MCLAG enhancements HLD mentioned at : MCLAG Enhancements HLD SONiC#596
How to verify it
Compile and verify MCLAG behavior.
Description for the changelog
The PR includes the ICCPD code changes for the MCLAG enhancements support.
This PR must work with submitted PR's in other sonic repositories which are listed below.

Swss: Mclag enhacements support code changes. sonic-swss#1331
swss-common: MCLAG Enhancements per HLD https://github.com/Azure/SONiC/pull/596 sonic-swss-common#405
Mclagsyncd: Mclag enhacements sonic-swss#1349
Utilities: mclag enhancements as per HLD at Azure/SONIC#596 sonic-utilities#1138
Mgmt-FW: mclag enhancements as per HLD at Azure/SONIC#596 sonic-mgmt-framework#59
Mgmt-Common: mclag enhancements as per HLD at Azure/SONIC#596 sonic-mgmt-common#25

rathnasabapathyv · 2020-11-11T01:54:03Z

Can you please resolve the conflicts

prvattem · 2020-11-11T02:17:46Z

src/iccpd/src/iccp_ifm.c

Are these code changes needed?
All the code in this function has been marked under "#if 0"

@prvattem: Thanks for the comment. The unused function iccp_parse_if_vlan_info_from_netlink is being removed.

prvattem · 2020-11-11T02:18:26Z

src/iccpd/src/iccp_ifm.c

Are the changes in this function needed?

@prvattem: Thanks for the comment. The unused function iccp_parse_if_vlan_info_from_netlink is being removed.

prvattem · 2020-11-11T03:35:11Z

src/iccpd/src/mlacp_link_handler.c

Is this unwanted functionality?

@prvattem: Yes, This flush is not required when ICCPd session is coming up. The unwanted flush is being commented.

prvattem · 2020-11-11T03:38:51Z

src/iccpd/src/mlacp_link_handler.c

Is this functionality going to get enabled later?

Thanks for the comment. No unused code is deleted

prvattem · 2020-11-11T03:42:49Z

src/iccpd/src/mlacp_link_handler.c

Is this functionality going to get enabled later?
References to this function are also commented.

Thanks for the comment. Yes with latest enhancements this code is not used. But the plan is to keep the change in case if required to enable back.

prvattem · 2020-11-11T04:08:45Z

src/iccpd/src/mlacp_link_handler.c

Please remove unwanted code.

Thanks for the comment. The code kept to enable in the future.

prvattem · 2020-11-12T07:31:20Z

Please resolve the conflicts, if any.

Praveen-Brcm · 2020-11-13T09:07:23Z

Please resolve the conflicts, if any.

Resolved the merge conflicts.

Praveen-Brcm · 2020-11-13T09:07:41Z

Please resolve the conflicts, if any.

Merged conflicts are resolved

Praveen-Brcm · 2020-11-18T03:55:34Z

@rathnasabapathyv , @prvattem
Merge conflicts are resolved and the review comments are addressed can you please provide approvals if not more comments.
Thanks, - Praveen.

gechiang · 2021-06-01T03:20:09Z

src/iccpd/src/scheduler.c

    return 1;

- recv_err:
+recv_err:


Please fix alignment

Thanks, Done.

gechiang · 2021-06-01T03:21:00Z

src/iccpd/src/scheduler.c

    goto accept_client;

- reject_client:
+reject_client:


fix alignment

Thanks, Done.

gechiang · 2021-06-01T03:21:09Z

src/iccpd/src/scheduler.c

    return MCLAG_ERROR;

- accept_client:
+accept_client:


fix alignment

Thanks, Done.

gechiang · 2021-06-01T03:22:14Z

src/iccpd/src/scheduler.c

    }

- out:
+out:


fix alignment

Thanks, Done.

gechiang · 2021-06-01T03:22:48Z

src/iccpd/src/scheduler.c

    }

- conn_fail:
+conn_fail:


fix alignment

Thanks, Done.

gechiang · 2021-06-01T03:22:54Z

src/iccpd/src/scheduler.c

        close(connFd);
    }
- conn_ok:
+conn_ok:


fix alignment

Thanks, Done.

gechiang · 2021-06-01T03:23:02Z

src/iccpd/src/scheduler.c

    }

- time_update:
+time_update:


fix alignment

Thanks, Done.

gechiang · 2021-06-01T03:23:09Z

src/iccpd/src/scheduler.c

    return 0;

- no_time_update:
+no_time_update:


fix alignment

Thanks, Done.

gechiang · 2021-06-01T05:27:48Z

src/iccpd/src/mlacp_sync_update.c

+            mlacp_peer_mlag_intf_delete_handler(csm, pif->name);
+
+            /* Delete remote interface info from STATE_DB */
+            if (csm)


This checkis not needed. line 92 already ensure cms is not NILL

Thanks, Done.!

gechiang · 2021-06-01T05:32:17Z

src/iccpd/src/mlacp_sync_update.c

-                                arp_entry->ifname, show_ip_str(arp_entry->ipv4_addr), mac_str);
-                return MCLAG_ERROR;
+                if (err != ICCP_NLE_SEQ_MISMATCH) {
+                    ICCPD_LOG_NOTICE(__FUNCTION__, "ARP add failure for %s %s %s, status %d",


An error should be logged as ERROR and not NOTICE right?

Thanks, Done.!

gechiang · 2021-06-01T05:32:51Z

src/iccpd/src/mlacp_sync_update.c

-                                arp_entry->ifname, show_ip_str(arp_entry->ipv4_addr), mac_str);
-                return MCLAG_ERROR;
+                if (err != ICCP_NLE_SEQ_MISMATCH) {
+                    ICCPD_LOG_NOTICE(__FUNCTION__, "ARP delete failure for %s %s %s, status %d",


Error condition should be logged with ERROR and not NOTICE.

Thanks, Done.!

gechiang · 2021-06-01T05:53:14Z

src/iccpd/src/mclagdctl/mclagdctl.c

    ret = EXIT_SUCCESS;

- mclagdctl_disconnect:
+mclagdctl_disconnect:


fix alignment please

Thanks, Done.

pettershao-ragilenetworks · 2021-07-17T07:42:13Z

@Praveen-Brcm can you help check why mclagsyncd socket sometimes receive close but actually iccpd just send some message to socket: "Connection lost" , seems mclagsyncd send fine, but receive exist some problem. is my local build miss some thing?[another device seems fine]
message.txt

ul 17 07:32:10.853588 sonic ERR swss#orchagent: :- addVlan: 4103: Failed to create VLAN Vlan1 vid:1
Jul 17 07:32:10.988675 sonic NOTICE iccpd#mclagsyncd: :- main: 81: MCLAGSYNCD processing mclag_cfg_tbl notifications
Jul 17 07:32:10.988803 sonic NOTICE iccpd#mclagsyncd: :- processMclagDomainCfg: 529: Key(mclag domain_id):1;  op:SET
Jul 17 07:32:10.988888 sonic NOTICE iccpd#mclagsyncd: :- processMclagDomainCfg: 545:  MCLAGSYNCD CFG Table Updates : Field source_ip, Value: 172.28.145.38  EntryExits:0
Jul 17 07:32:10.988888 sonic NOTICE iccpd#mclagsyncd: :- processMclagDomainCfg: 545:  MCLAGSYNCD CFG Table Updates : Field peer_link, Value: PortChannel2  EntryExits:0
Jul 17 07:32:10.988915 sonic NOTICE iccpd#mclagsyncd: :- processMclagDomainCfg: 545:  MCLAGSYNCD CFG Table Updates : Field peer_ip, Value: 172.28.145.36  EntryExits:0
Jul 17 07:32:10.988940 sonic NOTICE iccpd#mclagsyncd: :- processMclagDomainCfg: 671: mclagsycnd: domain cfg processing; mandatory args present; Domain [1] send to iccpd
Jul 17 07:32:10.988965 sonic NOTICE iccpd#mclagsyncd: :- processMclagDomainCfg: 678:  MCLAGSYNCD CFG Table Updates : domain_id:1 op_type:0 attrBmap:0x7 attrDelBmap:0x0 cfg_info.local_ip 172.28.145.38, peer_ip: 172.28.145.36 peer_link:PortChannel2 system_mac:58:69:6c:fb:21:38 session_timeout:0 keepalive_time:0
Jul 17 07:32:10.989030 sonic NOTICE iccpd#mclagsyncd: :- processMclagDomainCfg: 749:  MCLAGSYNCD CFG Table Updates: domain_id:1 infor_len:84 infor_start:0x55f12a5cc750
Jul 17 07:32:10.989030 sonic NOTICE iccpd#mclagsyncd: :- processMclagDomainCfg: 761: mclagsycnd send msg to iccpd, msg_len =84, msg_type =2    count : 0 ver = 1
Jul 17 07:32:10.989069 sonic NOTICE iccpd#mclagsyncd: :- processMclagDomainCfg: 772:  Add dependent selectables to select
Jul 17 07:32:10.989691 sonic NOTICE iccpd#mclagsyncd: :- addDomainCfgDependentSelectables: 780:  MCLAGSYNCD create state fdb table
Jul 17 07:32:10.990154 sonic NOTICE iccpd#iccpd: message repeated 81 times: [ [ICCP_FSM.NOTICE] csm null with peer ip [172.28.145.36]]
Jul 17 07:32:10.990154 sonic NOTICE iccpd#iccpd: [iccp_mclagsyncd_mclag_domain_cfg_handler.NOTICE] recv cfg msg ; domain_id:1 op_type:1 attr_bmap:0x7 local_ip:172.28.145.38 peer_ip:172.28.145.36 peer_ifname:PortChannel2 system_mac:58:69:6c:fb:21:38 session_timeout:0 keepalive_time:0
Jul 17 07:32:10.990154 sonic NOTICE iccpd#iccpd: [ICCP_FDB.NOTICE] mlacp_mac_msg_queue_reinit clear mac_msg_list pointers in existing MAC entries
Jul 17 07:32:10.990154 sonic NOTICE iccpd#iccpd: [ICCP_FSM.NOTICE] scheduler session disconnect handler
Jul 17 07:32:10.990231 sonic NOTICE iccpd#iccpd: [ICCP_FDB.NOTICE] mlacp_mac_msg_queue_reinit clear mac_msg_list pointers in existing MAC entries
Jul 17 07:32:10.992278 sonic NOTICE iccpd#mclagsyncd: :- addDomainCfgDependentSelectables: 783:  MCLAGSYNCD create state vlan member table
Jul 17 07:32:10.992744 sonic NOTICE iccpd#mclagsyncd: :- addDomainCfgDependentSelectables: 786:  MCLAGSYNCD create cfg mclag intf table
Jul 17 07:32:10.992744 sonic NOTICE iccpd#mclagsyncd: :- addDomainCfgDependentSelectables: 794:  MCLAGSYNCD Add state_fdb_tbl to selectable
Jul 17 07:32:10.992744 sonic NOTICE iccpd#mclagsyncd: :- addDomainCfgDependentSelectables: 801:  MCLAGSYNCD Add p_state_vlan_mbr_subscriber_table  to selectable
Jul 17 07:32:10.992744 sonic NOTICE iccpd#mclagsyncd: :- addDomainCfgDependentSelectables: 808: MCLagSYNCD Adding mclag_intf_cfg_tbl to selectable
Jul 17 07:32:10.993196 sonic INFO iccpd#supervisord: iccpd Connection lost, reconnecting...
Jul 17 07:32:10.996576 sonic NOTICE iccpd#mclagsyncd: :- mclagsyncdFetchSystemMacFromConfigdb: 78: mclagysncd: system_mac:58:69:6c:fb:21:38
Jul 17 07:32:10.996709 sonic INFO iccpd#supervisord: iccpd Waiting for connection...
Jul 17 07:32:11.637348 sonic NOTICE iccpd#iccpd: [ICCP_FSM.NOTICE] csm config error with peer ip [172.28.145.36]

root@sonic:/home/admin# docker exec -it iccpd bash
root@sonic:/# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 07:30 pts/0    00:00:00 /usr/bin/python /usr/bin/supervisord
root        17     1  0 07:30 pts/0    00:00:00 /usr/sbin/rsyslogd -n -iNONE
root        29     1  0 07:30 pts/0    00:00:00 bash /usr/bin/iccpd.sh
root        31    29  0 07:30 pts/0    00:00:00 mclagsyncd
root        32    29  0 07:30 pts/0    00:00:00 iccpd
root        52     0  0 07:49 pts/1    00:00:00 bash
root        57    52  0 07:49 pts/1    00:00:00 ps -ef
root@sonic:/# kill -9 31
root@sonic:/# kill -9 32
root@sonic:/# mclagsyncd &
[1] 58
root@sonic:/# Waiting for connection...

root@sonic:/# iccpd &
[2] 60
root@sonic:/# Connected!
Connection lost, reconnecting...
Waiting for connection...

root@sonic:/#

pettershao-ragilenetworks · 2021-07-21T02:36:01Z

server receive q's size is not empty, seems read() issue:

root@sonic:/home/admin# netstat -ap | grep 2626
tcp        0      0 127.0.0.6:2626          0.0.0.0:*               LISTEN      4692/mclagsyncd
tcp        0      0 localhost:52388         127.0.0.6:2626          ESTABLISHED 4693/iccpd
tcp      863      0 127.0.0.6:2626          localhost:52388         ESTABLISHED 4692/mclagsyncd

Praveen-Brcm · 2021-07-22T19:13:52Z

server receive q's size is not empty, seems read() issue:

root@sonic:/home/admin# netstat -ap | grep 2626
tcp        0      0 127.0.0.6:2626          0.0.0.0:*               LISTEN      4692/mclagsyncd
tcp        0      0 localhost:52388         127.0.0.6:2626          ESTABLISHED 4693/iccpd
tcp      863      0 127.0.0.6:2626          localhost:52388         ESTABLISHED 4692/mclagsyncd

@pettershao-ragilenetworks , The fix for this has been addressed part of PR# sonic-net/sonic-swss#1832 .? Thanks, - Prvaeen

Praveen-Brcm · 2021-07-22T21:08:00Z

@gechiang : Can this be merged now as the PR 1338 is merged, and PR 1331 is blocked on VS tests to re-run. Please confirm and merge.? Thanks - Praveen.

pettershao-ragilenetworks · 2021-07-23T00:42:38Z

server receive q's size is not empty, seems read() issue:
root@sonic:/home/admin# netstat -ap | grep 2626
tcp        0      0 127.0.0.6:2626          0.0.0.0:*               LISTEN      4692/mclagsyncd
tcp        0      0 localhost:52388         127.0.0.6:2626          ESTABLISHED 4693/iccpd
tcp      863      0 127.0.0.6:2626          localhost:52388         ESTABLISHED 4692/mclagsyncd
@pettershao-ragilenetworks , The fix for this has been addressed part of PR# Azure/sonic-swss#1832 .? Thanks, - Prvaeen

yes.

gechiang · 2021-07-23T01:00:37Z

@gechiang : Can this be merged now as the PR 1338 is merged, and PR 1331 is blocked on VS tests to re-run. Please confirm and merge.? Thanks - Praveen.

@Praveen-Brcm, @rlhui I think we discussed this in the meeting. This PR also needs (sonic-net/sonic-swss#1331) or else the existing MCLAG will be broken as soon as this PR is merged... So it was decided that we will also wait for the SWSS(1331) to complete and at that time merge both in parallel to reduce the feature broken time... Is this not what we agreed?

Praveen-Brcm · 2021-07-23T01:38:12Z

@gechiang : Can this be merged now as the PR 1338 is merged, and PR 1331 is blocked on VS tests to re-run. Please confirm and merge.? Thanks - Praveen.

@Praveen-Brcm, @rlhui I think we discussed this in the meeting. This PR also needs (Azure/sonic-swss#1331) or else the existing MCLAG will be broken as soon as this PR is merged... So it was decided that we will also wait for the SWSS(1331) to complete and at that time merge both in parallel to reduce the feature broken time... Is this not what we agreed?

@gechiang : Yes that was the agreement. Earlier we had the dependency with PR#1338 and we had to let that merge for the VS tests to pass for PR#1331. While we already merge 4 of the related PR's my request was to consider the current PR to merge as well, while we hope to have the PR #1331 merge soon once the VS tests pass.

gechiang · 2021-07-23T01:57:44Z

@gechiang : Can this be merged now as the PR 1338 is merged, and PR 1331 is blocked on VS tests to re-run. Please confirm and merge.? Thanks - Praveen.

@Praveen-Brcm, @rlhui I think we discussed this in the meeting. This PR also needs (Azure/sonic-swss#1331) or else the existing MCLAG will be broken as soon as this PR is merged... So it was decided that we will also wait for the SWSS(1331) to complete and at that time merge both in parallel to reduce the feature broken time... Is this not what we agreed?

@gechiang : Yes that was the agreement. Earlier we had the dependency with PR#1338 and we had to let that merge for the VS tests to pass for PR#1331. While we already merge 4 of the related PR's my request was to consider the current PR to merge as well, while we hope to have the PR #1331 merge soon once the VS tests pass.

IF the MC LAG feature in current master branch is already broken then we might as well go ahead get this merged... but if it is not, then I prefer to not break the feature for an unknown (potentially long) period of time...

haslersn · 2021-08-05T15:39:15Z

Is there a way to download an image (sonic-aboot-broadcom.swi) built from master? Otherwise, where do I get an image for testing MCLAG?

* MCLAG enhacements ICCPd initial code commit * Resolving the merge conflicts with orighin * L3 MCLAG Enhancements and Unique IP Changes. * Addressed review comments Co-authored-by: Tapash Das <tapash.das@broadcom.com>

akokhan · 2021-08-10T06:35:12Z

Is there a way to download an image (sonic-aboot-broadcom.swi) built from master? Otherwise, where do I get an image for testing MCLAG?

https://sonic-build.azurewebsites.net/ui/sonic/Pipelines

pettershao-ragilenetworks · 2021-08-10T08:38:16Z

src/iccpd/src/mlacp_link_handler.c

+                    ICCPD_LOG_DEBUG("ICCP_FDB", "MAC update from mclagsyncd: Update MAC remote to local %s, vlan %d"
+                            " ifname %s, del entry from MCLAG_FDB_TABLE",
+                            mac_addr_to_str(mac_msg->mac_addr), mac_msg->vid, mac_msg->ifname);
+                    del_mac_from_chip(mac_msg);


how do we distinguish this ADD is local learn or by "add back" action?

* MCLAG enhacements ICCPd initial code commit * Resolving the merge conflicts with orighin * L3 MCLAG Enhancements and Unique IP Changes. * Addressed review comments Co-authored-by: Tapash Das <tapash.das@broadcom.com>

sandyw777 · 2025-01-17T07:16:29Z

src/iccpd/src/mlacp_sync_update.c


-                            /*Update local item*/
-                            memcpy(&mac_msg->ifname, MacData->ifname, MAX_L_PORT_NAME);
+                            add_mac_to_chip(mac_msg, mac_msg->fdb_type);


@Praveen-Brcm
You changed the behavior from del_mac_from_chip() to add_mac_to_chip() even when the peerlink is down. Is it intended? Why?
Because the behavior is different from the behavior at line#531, the mac address is added when the peerlink is up.
Thank you!

sandyw777 · 2025-01-17T07:24:38Z

src/iccpd/src/mlacp_sync_update.c

+                        /*Redirect the mac to peer-link*/
+                         /*must redirect but if peerlink is down FdbOrch will delete MAC */
+                        memcpy(&mac_msg->ifname, csm->peer_itf_name, MAX_L_PORT_NAME);
+                        add_mac_to_chip(mac_msg, mac_msg->fdb_type);


@Praveen-Brcm
You add the mac even when the peerlink is down. Is it intended? Why?
Because the behavior is different from the behavior at line#531, the mac address is added when the peerlink is up.
Thank you!

sandyw777 · 2025-01-17T07:34:00Z

src/iccpd/src/mlacp_sync_update.c

@@ -347,11 +530,22 @@ int mlacp_fsm_update_mac_entry_from_peer( struct CSM* csm, struct mLACPMACData *
                if (csm->peer_link_if && csm->peer_link_if->state == PORT_STATE_UP)
                    add_mac_to_chip(mac_msg, mac_msg->fdb_type);


@Praveen-Brcm
The mac address is added only when peer-link is configured and the link is up.

Praveen-Brcm mentioned this pull request Jun 24, 2020

Mclag enhacements support code changes. sonic-net/sonic-swss#1331

Merged

gitsabari mentioned this pull request Jul 9, 2020

Mclag enhacements sonic-net/sonic-swss#1349

Merged

Praveen-Brcm mentioned this pull request Sep 22, 2020

iccpd service is not running in the image built from latest SONIC mainline code. #5310

Open

This was referenced Oct 30, 2020

MCLAG Enhancements Praveen-Brcm/sonic-swss-common#1

Closed

MCLAG Enhancements per HLD https://github.com/Azure/SONiC/pull/596 sonic-net/sonic-swss-common#405

Merged

rathnasabapathyv previously approved these changes Nov 11, 2020

View reviewed changes

prvattem reviewed Nov 11, 2020

View reviewed changes

Praveen-Brcm dismissed rathnasabapathyv’s stale review via e62594f November 12, 2020 08:24

Praveen-Brcm added 5 commits November 12, 2020 21:46

MCLAG enhacements ICCPd initial code commit

a110dac

MCLAG enhacements ICCPd code commit

69d6c49

Remove unused code.

bd8b3e3

Remove unused code.

d8139e9

Resolving the merge conflicts with orighin

4e78479

Praveen-Brcm force-pushed the mclag_enhacements branch from 5745928 to 4e78479 Compare November 13, 2020 08:56

Praveen-Brcm and others added 3 commits November 13, 2020 23:51

Removing unused re-definition code.

d608414

Fixing build issues.

32f1c4a

L3 MCLAG Enhancements and Unique IP Changes.

176ace0

lguohan force-pushed the master branch from 3690c1a to 512eb6b Compare December 25, 2020 18:34

Praveen-Brcm added 2 commits December 30, 2020 14:14

Merge branch 'master' into mclag_enhacements

1f9cc4a

Merge branch 'master' into mclag_enhacements

b15ec5d

gechiang reviewed Jun 1, 2021

View reviewed changes

jianjundong mentioned this pull request Jun 2, 2021

[MCLAG] Improve performance and fix some bugs #7769

Open

4 tasks

Praveen-Brcm added 2 commits June 13, 2021 18:02

Merge branch 'master' into mclag_enhacements

27e45d9

Addressed review comments

c2ac46b

Praveen-Brcm dismissed rathnasabapathyv’s stale review via c2ac46b June 14, 2021 01:06

gechiang approved these changes Aug 4, 2021

View reviewed changes

gechiang merged commit 82b6bcf into sonic-net:master Aug 4, 2021

pettershao-ragilenetworks reviewed Aug 10, 2021

View reviewed changes

yxieca mentioned this pull request Aug 10, 2021

[warm-reboot] APPLY_VIEW failure in set operation "SAI_API_FDB:brcm_sai_set_fdb_entry_attribute" #8367

Closed

vaibhavhd mentioned this pull request Aug 11, 2021

Fix warmboot issue PR##8367 sonic-net/sonic-swss#1866

Merged

judyjoseph added the Included in 202106 Branch label Aug 20, 2021

sandyw777 reviewed Jan 17, 2025

View reviewed changes

                   }
-               out:
+              out:

		@@ -347,11 +530,22 @@ int mlacp_fsm_update_mac_entry_from_peer( struct CSM* csm, struct mLACPMACData *
		if (csm->peer_link_if && csm->peer_link_if->state == PORT_STATE_UP)
		add_mac_to_chip(mac_msg, mac_msg->fdb_type);

Conversation

Praveen-Brcm commented Jun 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rathnasabapathyv commented Nov 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prvattem commented Nov 12, 2020

Uh oh!

Praveen-Brcm commented Nov 13, 2020

Uh oh!

Praveen-Brcm commented Nov 13, 2020

Uh oh!

Praveen-Brcm commented Nov 18, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Praveen-Brcm commented Jun 19, 2020 •

edited

Loading

pettershao-ragilenetworks commented Jul 17, 2021 •

edited

Loading

pettershao-ragilenetworks commented Jul 21, 2021 •

edited

Loading