[config]Static routes to config_db by d-dashkov · Pull Request #1534 · sonic-net/sonic-utilities

d-dashkov · 2021-03-30T15:33:40Z

What I did
fixes sonic-net/sonic-buildimage#6997
Added process for writing static routes to config_db.

fixes sonic-net/sonic-buildimage#7261
Added interface validation before creating route.

How I did it
Refactored add/del route functions. Wrote generic function that returns a vtysh command and an entry for config_db with name.

sudo config route add prefix 1.2.3.4/32 nexthop 30.0.0.5
sudo config route add prefix 1.3.3.4/32 nexthop 30.0.0.5
sudo config route add prefix 1.4.3.4/32 nexthop 30.0.0.5
sudo config route add prefix vrf Vrf-RED 1.5.3.4/32 nexthop 30.0.0.5
sudo config save -y
sudo reboot

Instead of running the vtysh command, it creates config_db entries that will be loaded by the ffr daemon:

    "STATIC_ROUTE": {
        "1.2.3.4/32": {
            "nexthop": "30.0.0.5"
        },
        "1.3.3.4/32": {
            "nexthop": "30.0.0.5"
        },
        "1.4.3.4/32": {
            "nexthop": "30.0.0.5"
        },
        "Vrf-RED|1.5.3.4/32": {
            "nexthop": "30.0.0.5"
        }
    }

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

prsunny · 2021-03-30T16:32:56Z

Here, the user is expected to run route update after every reboot?

d-dashkov · 2021-03-30T16:41:28Z

@prsunny
I don't think so, user shouldn't run this command manually after reboot. But I really don't know where this update function can be called, because it should be executed after the RIFs are generated, and running it in the startup script shouldn't work.

prsunny · 2021-03-30T17:00:47Z

@prsunny
I don't think so, user shouldn't run this command manually after reboot. But I really don't know where this update function can be called, because it should be executed after the RIFs are generated, and running it in the startup script shouldn't work.

ok, so how is it expected to be restored from config_db after reboot?

d-dashkov · 2021-03-30T19:13:35Z

@prsunny
I don't think so, user shouldn't run this command manually after reboot. But I really don't know where this update function can be called, because it should be executed after the RIFs are generated, and running it in the startup script shouldn't work.

ok, so how is it expected to be restored from config_db after reboot?

I'm not sure, but maybe the update can be triggered using crontab or along with running the bgp container.

prsunny · 2021-03-30T23:08:44Z

Can you add unit-tests as well?

prsunny · 2021-03-30T23:11:10Z

Why is the schema showing nexthop in key?

       "1.2.3.4/32|30.0.0.5": {
            "nexthop": "30.0.0.5",
            "prefix": "1.2.3.4/32"
        },

caizhenghui-juniper · 2021-03-31T01:24:11Z

And also why do we need prefix in the value field? It's already part of the key...
For static route table the scheme is prefix as key and nexthop as field as the simplest case..

caizhenghui-juniper · 2021-03-31T01:28:13Z

we already have frrcfgd monitoring config db, so the part of programming frr cfg is not necessary as long as config_db is updated, isn't it?

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

d-dashkov · 2021-03-31T17:52:23Z

@prsunny @caizhenghui-juniper
I updated the record name with a prefix as the key and nexthop as the value, and also added base UT. But I cannot find frrcfgd in the repositories, can you tell me where it is?

caizhenghui-juniper · 2021-03-31T18:02:17Z

Hi, Dmytro, Here it is: https://github.com/Azure/sonic-buildimage/tree/master/src/sonic-frr-mgmt-framework/frrcfgd One more issue: in swss, we do require VRF name conforms with prefix Vrf, so please correct that in VRF test case accordingly. Small problem 😊 And the schema defined in frrcfgd for vrf is using vrf_name instead of vrf. Plus, are you still setting prefix as field? At line 3248? Thanks Zhenghui From: Dmytro ***@***.***> Reply-To: Azure/sonic-utilities ***@***.***> Date: Wednesday, March 31, 2021 at 1:52 PM To: Azure/sonic-utilities ***@***.***> Cc: Zhenghui Cai ***@***.***>, Mention ***@***.***> Subject: Re: [Azure/sonic-utilities] [config]Static routes to config_db (#1534) [External Email. Be cautious of content] @prsunny<https://github.com/prsunny> @caizhenghui-juniper<https://github.com/caizhenghui-juniper> I updated the record name with a prefix as the key and nexthop as the value, and also added base UT. But I cannot find frrcfgd in the repositories, can you tell me where it is? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1534 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKDUQJVEG3JZD3HB5MP773DTGNOOTANCNFSM42CKDI4Q>. Juniper Business Use Only

caizhenghui-juniper · 2021-03-31T18:34:21Z

and can you put support for ECMP too? With more than 1 next hops for the static routes?

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

shi-su · 2021-04-05T16:22:10Z

tests/static_routes_test.py

+        ["prefix", "3.2.3.4/32", "nexthop", "vrf", "Vrfred", "30.0.0.6"], obj=obj)
+        print(result.exit_code, result.output)
+        assert ('3.2.3.4/32') in db.cfgdb.get_table('STATIC_ROUTE')
+        assert db.cfgdb.get_entry('STATIC_ROUTE', '3.2.3.4/32') == {"nexthop": "30.0.0.6", "nexthop_vrf": "Vrfred"}


I think we need to follow a certain schema design for the STATIC_ROUTE table. In sonic-net/SONiC#585, there is a schema for the table but it seems different here.

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

d-dashkov · 2021-04-06T19:17:31Z

I'm not sure if this is a bug or not, but after reboot, the frr daemon does not start with the system, and also after manual start, the daemon sees a static route, but does not add it to frr cfg

Before reboot

sudo config route add prefix 1.2.3.4/32 nexthop 30.0.0.5

"STATIC_ROUTE": {
	"1.2.3.4/32": {
		"nexthop": "30.0.0.5"
	}
}


admin@sonic:~$ redis-cli -n 4 keys "*" | grep STATIC_ROUTE
STATIC_ROUTE|1.2.3.4/32
admin@sonic:~$ redis-cli -n 4 hgetall "STATIC_ROUTE|1.2.3.4/32"
1) "nexthop"
2) "30.0.0.5"

admin@sonic:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [200/0] via 192.168.111.3, eth0, weight 1, 00:05:24
S>  1.2.3.4/32 [1/0] via 30.0.0.7 (recursive), weight 1, 00:00:06
C>* 10.1.0.1/32 is directly connected, Loopback0, 00:05:07
C>* 192.168.111.0/24 is directly connected, eth0, 00:05:25

After reboot

Syslog

May  7 21:22:18.776046 sonic DEBUG bgp#/frrcfgd: entering BGP configuration daemon
May  7 21:22:18.780913 sonic DEBUG bgp#/frrcfgd: send initial enable command to bgpd
May  7 21:22:18.784148 sonic DEBUG bgp#/frrcfgd: send initial enable command to zebra
May  7 21:22:18.784148 sonic DEBUG bgp#/frrcfgd: send initial enable command to staticd
May  7 21:22:18.784148 sonic DEBUG bgp#/frrcfgd: send initial enable command to bfdd
May  7 21:22:18.784148 sonic DEBUG bgp#/frrcfgd: send initial enable command to ospfd
May  7 21:22:18.784148 sonic DEBUG bgp#/frrcfgd: send initial enable command to pimd
May  7 21:22:18.810021 sonic DEBUG bgp#/frrcfgd: entering VTYSH proxy thread
May  7 21:22:18.810021 sonic DEBUG bgp#/frrcfgd: waiting for client connection ...
May  7 21:22:18.899398 sonic DEBUG bgp#/frrcfgd: Init Cached DB data
May  7 21:22:19.926529 sonic DEBUG bgp#/frrcfgd:   STATIC_ROUTE&&1.2.3.4/32 : {'nexthop': '30.0.0.5'}

admin@sonic:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [200/0] via 192.168.111.3, eth0, weight 1, 00:05:24
C>* 10.1.0.1/32 is directly connected, Loopback0, 00:05:07
C>* 192.168.111.0/24 is directly connected, eth0, 00:05:25

And if I try to change the nexthop on this entry, staticd crashes because it has no previous entry

sudo config route add prefix 1.2.3.4/32 nexthop 30.0.0.5

May  7 21:09:27.483443 sonic DEBUG bgp#/frrcfgd: ----------------------------------
May  7 21:09:27.483607 sonic DEBUG bgp#/frrcfgd:  BGP table handling
May  7 21:09:27.483690 sonic DEBUG bgp#/frrcfgd: ----------------------------------
May  7 21:09:27.483764 sonic DEBUG bgp#/frrcfgd: table : STATIC_ROUTE
May  7 21:09:27.483881 sonic DEBUG bgp#/frrcfgd: key   : 1.2.3.4/32
May  7 21:09:27.483958 sonic DEBUG bgp#/frrcfgd: op    : SET
May  7 21:09:27.484074 sonic DEBUG bgp#/frrcfgd: data  :
May  7 21:09:27.484152 sonic DEBUG bgp#/frrcfgd:         nexthop    - 30.0.0.7
May  7 21:09:27.484225 sonic DEBUG bgp#/frrcfgd:
May  7 21:09:27.484326 sonic INFO bgp#/frrcfgd: value for table STATIC_ROUTE prefix default key 1.2.3.4/32 changed to {'nexthop': (30.0.0.7, UPDATE)}
May  7 21:09:27.484439 sonic INFO bgp#/frrcfgd: Set static IP route for vrf default prefix 1.2.3.4/32
May  7 21:09:27.484525 sonic DEBUG bgp#/frrcfgd: execute command vtysh -c 'configure terminal' -c 'vrf default' -c 'no ip route 1.2.3.4/32  30.0.0.5     ' for table STATIC_ROUTE.
May  7 21:09:27.484646 sonic DEBUG bgp#/frrcfgd: VTYSH CMD: configure terminal daemons: ['staticd']
May  7 21:09:27.484725 sonic DEBUG bgp#/frrcfgd: VTYSH CMD: vrf default daemons: ['staticd']
May  7 21:09:27.485096 sonic DEBUG bgp#/frrcfgd: VTYSH CMD: no ip route 1.2.3.4/32  30.0.0.5 daemons: ['staticd']
May  7 21:09:27.485581 sonic DEBUG bgp#/frrcfgd: [staticd] command return code: 1
May  7 21:09:27.485777 sonic DEBUG bgp#/frrcfgd:
May  7 21:09:27.485883 sonic DEBUG bgp#/frrcfgd: VTYSH CMD: end daemons: ['staticd']
May  7 21:09:27.485995 sonic ERR bgp#/frrcfgd: command execution failure. Command: "vtysh -c 'configure terminal' -c 'vrf default' -c 'no ip route 1.2.3.4/32  30.0.0.5     '"
May  7 21:09:27.486073 sonic ERR bgp#/frrcfgd: failed running FRR command: no ip route 1.2.3.4/32  30.0.0.5
May  7 21:09:27.486161 sonic ERR bgp#/frrcfgd: failed running static route config command
May  7 21:09:27.486235 sonic DEBUG bgp#/frrcfgd: ignore cache update for nexthop because of STAT_FAIL
May  7 21:09:27.486308 sonic INFO bgp#/frrcfgd: delete table row STATIC_ROUTE&&default|1.2.3.4/32 from cache

prsunny

I see that this PR is missing ecmp handling. In the existing code, by calling vtysh, it does handle ECMP. My suggestion is, when user configure the same route prefix with a different nexthop, it should modify the STATIC_ROUTE DB entry instead of overwriting thus supporting ECMP (similar to vtysh). @shi-su, this should be handled by bgpcfgd.

prsunny · 2021-04-06T21:11:19Z

tests/static_routes_test.py

+        runner = CliRunner()
+        obj = {'config_db':db.cfgdb}
+
+        # config route add prefix 2.2.3.4/32 nexthop 30.0.0.6


please update the comment to include vrf

prsunny · 2021-04-06T21:12:10Z

tests/static_routes_test.py

+        runner = CliRunner()
+        obj = {'config_db':db.cfgdb}
+
+        # config route add prefix 3.2.3.4/32 nexthop 30.0.0.6


same here, please update the correct command with vrf

prsunny · 2021-04-06T21:13:24Z

config/main.py

-    clicommon.run_command(cmd)
+    key, route = cli_sroute_to_config(ctx, command_str)
+    config_db = ctx.obj['config_db']
+    config_db.set_entry("STATIC_ROUTE", key, route)


If user update the route, this would overwrite the entry, correct?

prsunny · 2021-04-06T21:21:23Z

Please update CLI command reference document

shi-su · 2021-04-06T22:27:05Z

I see that this PR is missing ecmp handling. In the existing code, by calling vtysh, it does handle ECMP. My suggestion is, when user configure the same route prefix with a different nexthop, it should modify the STATIC_ROUTE DB entry instead of overwriting thus supporting ECMP (similar to vtysh). @shi-su, this should be handled by bgpcfgd.

Yes, the STATIC_ROUTE DB entry needs to be updated when nexthop changes. The bgpcfgd should be able to know the changes and only apply the differences for the same route prefix.

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

prsunny · 2021-04-12T23:25:16Z

config/main.py

-            if vrf_name == "":
-                cmd += ' {} nexthop-vrf {}'.format(ip, vrf_name_dst)
+    config_db = ctx.obj['config_db']
+    key, route = cli_sroute_to_config(ctx, command_str)


Now that we are writing to DB, lets check the correctness of key (like a valid ip prefix)

prsunny · 2021-04-12T23:26:47Z

config/main.py

+        if 'ifname' in current_entry:
+            ifname = current_entry['ifname'].split(',')
+
+        nh_zip = list(itertools.zip_longest(nh, nh_vrf, ifname, fillvalue=''))


can you add some comments to understand the logic in this section?

prsunny · 2021-04-12T23:27:42Z

config/main.py

+            ctx.fail('Not found {} in {}'.format(cmd_tuple, key))
+
+        if len(nh) == 0 or (len(nh) == 1 and nh[0] == ''):
+            config_db.set_entry("STATIC_ROUTE", key, None)


if user just deletes the route without mentioning any nexthops, then the route is expected to be fully deleted.

If the user tries to delete a route without nexthop (for example, 'сonfig del config route del prefix 1.2.3.4/32'), he will receive an error: 'argument is not in pattern'. So if the user has previously added ECMP routes, he must remove them one by one.
This condition checks if the record still has any nexthop, if so, it just updates this record, if it doesn't have the nexthop, it removes the key. Is this the correct behavior or should user be able to delete the entire entry at once?

IMO, user should be able to delete the entire route at once.

prsunny · 2021-04-12T23:31:50Z

tests/static_routes_test.py

+        print(result.exit_code, result.output)
+        assert not ('3.2.3.4/32') in db.cfgdb.get_table('STATIC_ROUTE')
+
+    def test_several_nexthops_static_route(self):


please rename it to test_multiple_nexthops_static_route

prsunny · 2021-04-12T23:42:03Z

tests/static_routes_test.py

+        print(result.exit_code, result.output)
+        assert not ('11.2.3.4/32') in db.cfgdb.get_table('STATIC_ROUTE')
+
+    def test_static_route_ECMP_nextfop_vrf(self):


nexthop - typo

prsunny · 2021-04-12T23:42:41Z

tests/static_routes_test.py

+        assert ('6.2.3.4/32') in db.cfgdb.get_table('STATIC_ROUTE')
+        assert db.cfgdb.get_entry('STATIC_ROUTE', '6.2.3.4/32') == {"nexthop": "30.0.0.6,30.0.0.7"}
+
+        # config route del prefix 6.2.3.4/32 nexthop 30.0.0.7


please have a test to del prefix without any nexthops and ensure its removed from DB.

prsunny · 2021-04-23T02:17:38Z

config/main.py

+    key, route = cli_sroute_to_config(ctx, command_str)
+
+    # If defined intf name, check if it exists
+    if 'ifname' in route:


As commented by @arlakshm for PR #1535 , please check if ifname belongs to VLAN_INTERFACE, INTERFACE and PORTCHANNEL_INTERFACE. If not return failure as "IP interface doesn't exist"

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

config/main.py

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

arlakshm · 2021-04-29T21:57:59Z

config/main.py

+    config_db = ConfigDBConnector()
+    config_db.connect()
+    ctx.obj = {}
+    ctx.obj['config_db'] = config_db


can you add the support for multi asic ?
Similar to how it is done for interface command https://github.com/Azure/sonic-utilities/blob/08337aa7637b290bb8407c38b2a5dbe3e8383b3e/config/main.py#L2359

@arlakshm, can we add multiasic as another enhancement PR?

@prsunny, Sure. Can we create an issue or task to track this?

Created issue - #1608

shi-su

I do not have any other comments. Please check with other reviewers.

prsunny · 2021-05-05T01:19:30Z

config/main.py

+        else:
+            route['distance'] = '0'
+
+        if 'blackhole' in route:


We would need to handle one scenario for blackhole routes. If the user configure with "ifname" as "null", could you please add the attribute 'blackhole' as true.

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

shi-su · 2021-05-07T19:38:27Z

config/main.py

        else:
-            route['blackhole'] = 'false'
+            # If the user configure with "ifname" as "null", set 'blackhole' attribute as true.
+            if 'ifname' in route and route['ifname'] == 'null':


Just want to confirm. Does this mean that it is necessary to confirm the route is completely removed before setting it to blackhole? If a route already exists in Config_DB and then we configure it to blackhole, the command would simply get ignored? If this is the case, could you please add logic to notify the caller that the command is getting ignored (e.g., fail the command with an error message)?

Yes, it will be ignored if some entries already exist in the db. @prsunny can this suggestion will be add in #1608?

dprital · 2021-05-30T15:33:57Z

@prsunny - Can you please add the tag "Required to 202012" ?

* Write static routes to config_db * Update configuration with "ifname" as "null" Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

haslersn · 2021-07-03T20:25:36Z

Is this already available in the prebuilt sonic-aboot-broadcom.swi ? If not, then how to configure (and persist across reboots) static routes?

7f50b9815e14d90c02d9dce63fd08d90e25cee3f (HEAD -> 201911, origin/201911) handled update() function of fdb orchagent for FDB FLUSH event (sonic-net#1534) 17adc13b6ca21846fe27c94d6a16f9909c712d77 Add a check for warm-restart, and do a clear only when warm-restart is enable. (sonic-net#1498) d097260a5aa7bd611babd5062e220056374e23d8 Fixed compilation failure with debug option (sonic-net#1518) Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

d-dashkov added 4 commits March 30, 2021 17:15

Write static routes to config_db

47696b9

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

Update route name for db

7de1946

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

Resolve conflicts

eb87991

Tab replaced with spaces

93220d9

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

prsunny requested a review from shi-su March 30, 2021 23:08

Renamed entry and initial UT

cab6c7a

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

Added VRF prefix check and renamed VRF entry

fbff2e4

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

shi-su reviewed Apr 5, 2021

View reviewed changes

d-dashkov added 2 commits April 6, 2021 19:07

Depricated frr cfg

1a02681

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

Updated UT

6392385

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

d-dashkov requested a review from shi-su April 6, 2021 19:24

prsunny reviewed Apr 6, 2021

View reviewed changes

d-dashkov added 3 commits April 12, 2021 16:51

Initial ECMP

91723c8

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

Fix tests

091082b

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

Update doc

bc26317

prsunny reviewed Apr 12, 2021

View reviewed changes

prsunny reviewed Apr 23, 2021

View reviewed changes

prsunny mentioned this pull request Apr 23, 2021

Fix remove ip rif #1535

Merged

Update entry

319a3ea

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

d-dashkov dismissed prsunny’s stale review via 319a3ea April 28, 2021 13:58

shi-su reviewed Apr 28, 2021

View reviewed changes

config/main.py Outdated Show resolved Hide resolved

Fix command with interface

bc14735

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

arlakshm reviewed Apr 29, 2021

View reviewed changes

shi-su reviewed May 3, 2021

View reviewed changes

prsunny reviewed May 5, 2021

View reviewed changes

andyivanov previously approved these changes May 5, 2021

View reviewed changes

Update configuration with "ifname" as "null"

fb6d0eb

Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

d-dashkov dismissed andyivanov’s stale review via fb6d0eb May 7, 2021 10:07

shi-su reviewed May 7, 2021

View reviewed changes

prsunny mentioned this pull request May 11, 2021

[Static route] Add Multiasic support #1608

Open

prsunny approved these changes May 11, 2021

View reviewed changes

arlakshm approved these changes May 11, 2021

View reviewed changes

shi-su approved these changes May 13, 2021

View reviewed changes

prsunny merged commit ad801bf into sonic-net:master May 13, 2021

ppikh mentioned this pull request May 18, 2021

Impossible to add static route via interface sonic-net/sonic-buildimage#7643

Open

d-dashkov mentioned this pull request May 18, 2021

[config] Update unit tests for static routes #1619

Closed

volodymyrsamotiy added the Request for 202012 Branch label Jun 1, 2021

qiluo-msft pushed a commit that referenced this pull request Jun 3, 2021

[config]Static routes to config_db (#1534)

d096ff7

* Write static routes to config_db * Update configuration with "ifname" as "null" Signed-off-by: d-dashkov <Dmytro_Dashkov@Jabil.com>

qiluo-msft added the Included in 202012 Branch label Jun 3, 2021

vivekrnv mentioned this pull request Jun 30, 2021

[bgpcfgd] Couldn't configure a Static Route without BGP_ASN field sonic-net/sonic-buildimage#8027

Closed

DavidZagury mentioned this pull request Oct 6, 2021

Can't add route with a nexthop of a PortChannel sonic-net/sonic-buildimage#8907

Closed

anders-nexthop mentioned this pull request Jun 9, 2025

Issue #22420: Modify 'config route add' command not to include empty … #3862

Merged

Conversation

d-dashkov commented Mar 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prsunny commented Mar 30, 2021

Uh oh!

d-dashkov commented Mar 30, 2021

Uh oh!

prsunny commented Mar 30, 2021

Uh oh!

d-dashkov commented Mar 30, 2021

Uh oh!

prsunny commented Mar 30, 2021

Uh oh!

prsunny commented Mar 30, 2021

Uh oh!

caizhenghui-juniper commented Mar 31, 2021

Uh oh!

caizhenghui-juniper commented Mar 31, 2021

Uh oh!

d-dashkov commented Mar 31, 2021

Uh oh!

caizhenghui-juniper commented Mar 31, 2021 via email

Uh oh!

caizhenghui-juniper commented Mar 31, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-dashkov commented Apr 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prsunny left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prsunny commented Apr 6, 2021

Uh oh!

shi-su commented Apr 6, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shi-su left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-dashkov commented Mar 30, 2021 •

edited

Loading

d-dashkov commented Apr 6, 2021 •

edited

Loading