Configure midplane interface in multi-asic namespaces on a chassis

[mlorrillere]

This change is to create an eth1 interface in multi-asic namespaces on a linecard so that containers can connect to CHASSIS_APP_DB

Why I did it

Multi-asic instances running on a linecard can't connect to CHASSIS_APP_DB (running on the supervisor) because they are bridged using default docker bridge and can only access local linecard resources.

How I did it

• chassisdb.conf on a linecard will define midplane_subnet which is the subnet used for the internal network
• midplane_subnet is chosen by the vendor. The only restriction is that should be a /16 subnet. chassisdb_address in chassisdb.conf should be part of midplane_subnet.
• sonic-platform-modules-* is to create an eth1-midplane interface on the linecard that can connect to the supervisor. The IP assigned to eth1-midplane should be part of midplane_subnet and in the form X.Y.<slot id>.1
• interfaces-config.sh will create for each namespace of a multi-asic linecard (defined as NUM_ASIC > 1 and midplane_subnet present) an ipvlan eth1 interface that is attached to eth1-midplane. This interface is assigned an IP in the range defined by midplane_subnet (i.e. X.Y.<slot id>.(10 + namespace id))

To allow using a midplane_subnet in the 127.X.Y.Z/16 range, we change the loopback address of the namespace lo interface to 127.0.0.1/16 (default is /8). This step is already done on the host as per PR5353.

When midplane_subnet is in the 127.X.Y.Z/16 range, we enable routing on local network so that 127/16 can be used.

How to verify it

Loopback address of a namespace changed to 127.0.0.1/16:

root@sonic:~# sonic-netns-exec asic0 ip addr show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/16 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

New eth1 interface of asic0 namespace on a linecard in slot 6:

root@sonic:~# sonic-netns-exec asic0 ip addr show dev eth1
11: eth1@Bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d4:af:f7:2f:fe:39 brd ff:ff:ff:ff:ff:ff
    inet 127.100.6.10/16 scope host eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::d4af:f700:12f:fe39/64 scope link
       valid_lft forever preferred_lft forever

New eth1 interface of asic1 namespace on a linecard in slot 6:

root@sonic:~# sonic-netns-exec asic1 ip addr show dev eth1
12: eth1@Loopback0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d4:af:f7:2f:fe:39 brd ff:ff:ff:ff:ff:ff
    inet 127.100.6.11/16 scope host eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::d4af:f700:22f:fe39/64 scope link
       valid_lft forever preferred_lft forever

Check connectivity from namespace asic0 to CHASSIS_APP_DB:

root@sonic:~# sonic-db-cli -n asic0 CHASSIS_APP_DB PING
True

Additional outputs from ip route and arp commands:

ip route output from the linecard:

root@sonic:~# ip route
10.240.78.0/25 dev eth0 proto kernel scope link src 10.240.78.55
127.100.0.0/16 dev eth1-midplane proto kernel scope link src 127.100.6.1
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1

ip route output from namespace asic0:

root@sonic:~# sonic-netns-exec asic0 ip route
default via 240.127.1.1 dev eth0 metric 3523215360
127.100.0.0/16 dev eth1 proto kernel scope link src 127.100.6.10
240.127.1.0/24 dev eth0 proto kernel scope link src 240.127.1.3

ip route output from namespace asic1:

root@sonic:~# sonic-netns-exec asic1 ip route
default via 240.127.1.1 dev eth0 metric 3523215360
127.100.0.0/16 dev eth1 proto kernel scope link src 127.100.6.11
240.127.1.0/24 dev eth0 proto kernel scope link src 240.127.1.2

arp entries learned on linecard:

root@sonic:~# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
240.127.1.2              ether   02:42:f0:7f:01:02   C                     docker0
10.240.78.1              ether   98:5d:82:8f:ea:8b   C                     eth0
redis_chassis.server     ether   d4:af:f7:2f:fe:38   C                     eth1-midplane
240.127.1.3              ether   02:42:f0:7f:01:03   C                     docker0

arp entries learned on supervisor (arp entries of docker containers removed for clarity):

root@sonic:~# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
127.100.6.10             ether   d4:af:f7:2f:fe:39   C                     br1
127.100.6.1              ether   d4:af:f7:2f:fe:39   C                     br1
127.100.6.11             ether   d4:af:f7:2f:fe:39   C                     br1
10.240.78.1              ether   98:5d:82:8f:ea:8b   C                     eth0

Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006
• 202012
• 202106

Description for the changelog

Configure midplane interface in multi-asic namespaces on a chassis

[judyjoseph]

Thanks for the PR. Could you update the PR description with more details like ip route o/p in namespace/host , arp entries learned etc on a linecard as well as supervisor card.

[judyjoseph]

@arlakshm @abdosi @minionatwork could you take a look as well

[mlorrillere]

@judyjoseph @minionatwork

Thanks for the PR. Could you update the PR description with more details like ip route o/p in namespace/host , arp entries learned etc on a linecard as well as supervisor card.

Description updated with requested output.

[sudurais]

Looks good to me.

[judyjoseph]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mlok-nokia]

It looks good to me

[judyjoseph]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[anamehra]

Is there a particular reason the midplane interface name has to be 'eth1-midplane'? Could this be platform defined in chassisd.conf and sourced in interfaces-config.sh?