Skip to content

"libteamdctl: teamdctl_connect: Failed to connect using all CLIs" after a range of consecutive creation/deletion an interface portchannel #4129

@MaxYaremchuk

Description

@MaxYaremchuk

Description
This BUG is reproduced intermittently.
After a range of consecutive creation/deletion an interface portchannel teamd fails.

Steps to reproduce the issue:

  1. Create a portchannel
    config portchannel add PortChannel0001
  2. Add members to PortChannel0001
    config portchannel member add PortChannel0001 Ethernet0
    config portchannel member add PortChannel0001 Ethernet4
  3. Assign IP address on PortChannel0001
    config interface ip add PortChannel0001 192.168.100.1
  4. Remove IP address from PortChannel0001
    config interface ip remove $lag_name 192.168.100.1
  5. Remove members from PortChannel0001
    config portchannel member del PortChannel0001 Ethernet0
    config portchannel member del PortChannel0001 Ethernet4
  6. Delete PortChannel0001
    config portchannel del PortChannel0001
  7. Perform steps 1-6 100 times
  8. Create a portchannel
    config portchannel add PortChannel0001
  9. Add members to PortChannel0001
    config portchannel member add PortChannel0001 Ethernet0
    config portchannel member add PortChannel0001 Ethernet4
  10. Assign IP address on PortChannel0001
    config interface ip add PortChannel0001 192.168.100.2

Describe the results you received:

  1. PortChannel0001 is not listed in "show interface status" output
  2. Although PortChannel0001 is pingable teamdctl returns error
#teamdctl PortChannel0001 state dump  returns 
libteamdctl: teamdctl_connect: Failed to connect using all CLIs.
teamdctl_connect failed (Invalid argument)

Untitled

Interfaces in linux

# ifconfig
PortChannel0001: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9100
        inet 192.168.100.2  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::268a:7ff:fe7e:4f80  prefixlen 64  scopeid 0x20<link>
        ether 24:8a:07:7e:4f:80  txqueuelen 1000  (Ethernet)
        RX packets 4  bytes 852 (852.0 B)
        RX errors 0  dropped 4  overruns 0  frame 0
        TX packets 4  bytes 520 (520.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Proceses in teamd container:

root@sonic:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.2  58204 20432 pts/0    Ss+  14:37   0:01 /usr/bin/python /usr/bin/supervisord
root        13  0.0  0.1  41152 13464 pts/0    S    14:37   0:00 python /usr/bin/supervisor-proc-exit-listener
root        19  0.0  0.0 254352  3144 pts/0    Sl   14:37   0:00 /usr/sbin/rsyslogd -n
root        24  0.0  0.0 102888  4480 pts/0    Sl   14:37   0:00 /usr/bin/teammgrd
root        27  0.0  0.0 108944  4760 pts/0    Sl   14:37   0:00 /usr/bin/teamsyncd
root        34  0.0  0.0  18180  3384 pts/1    Ss   14:52   0:00 bash
root        42  0.0  0.0  36632  2780 pts/1    R+   14:52   0:00 ps aux

Describe the results you expected:
Interface PortChannel0001 should be present and teamdctl PortChannel0001 state dump should return PortChannel0001 state.

Additional information you deem important (e.g. issue happens only occasionally):
This BUG is reproduced intermittently.

Output of show version:

SONiC Software Version: SONiC.master.0-dirty-20200114.131506
Distribution: Debian 9.11
Kernel: 4.9.0-9-2-amd64
Build commit: 952dfb15
Build date: Tue Jan 14 13:28:50 UTC 2020
Built by: volodymyrs@r-build-sonic03

Platform: x86_64-mlnx_msn2100-r0
HwSKU: ACS-MSN2100
ASIC: mellanox
Serial Number: MT1646X05314
Uptime: 09:43:29 up 55 min,  1 user,  load average: 3.34, 3.54, 3.55

Docker images:
REPOSITORY                    TAG                              IMAGE ID            SIZE
docker-platform-monitor       latest                           aabdbf5b55b9        569MB
docker-platform-monitor       master.0-dirty-20200114.131506   aabdbf5b55b9        569MB
docker-sflow                  latest                           acc6d9a905f3        306MB
docker-sflow                  master.0-dirty-20200114.131506   acc6d9a905f3        306MB
docker-fpm-frr                latest                           2e17fdf8f565        325MB
docker-fpm-frr                master.0-dirty-20200114.131506   2e17fdf8f565        325MB
docker-syncd-mlnx             latest                           09df8b1f23d7        377MB
docker-syncd-mlnx             master.0-dirty-20200114.131506   09df8b1f23d7        377MB
docker-orchagent              latest                           6729c2e779af        323MB
docker-orchagent              master.0-dirty-20200114.131506   6729c2e779af        323MB
docker-teamd                  latest                           a710f94fa26c        305MB
docker-teamd                  master.0-dirty-20200114.131506   a710f94fa26c        305MB
docker-snmp-sv2               latest                           e4c726145422        339MB
docker-snmp-sv2               master.0-dirty-20200114.131506   e4c726145422        339MB
docker-lldp-sv2               latest                           d67cfab77277        303MB
docker-lldp-sv2               master.0-dirty-20200114.131506   d67cfab77277        303MB
docker-sonic-mgmt-framework   latest                           544660d56163        330MB
docker-sonic-mgmt-framework   master.0-dirty-20200114.131506   544660d56163        330MB
docker-sonic-telemetry        latest                           7df4698f004b        343MB
docker-sonic-telemetry        master.0-dirty-20200114.131506   7df4698f004b        343MB
docker-sonic-restapi          latest                           6834d43af7fb        293MB
docker-sonic-restapi          master.0-dirty-20200114.131506   6834d43af7fb        293MB
docker-router-advertiser      latest                           2fa815e41d75        282MB
docker-router-advertiser      master.0-dirty-20200114.131506   2fa815e41d75        282MB
docker-database               latest                           681324505d5c        282MB
docker-database               master.0-dirty-20200114.131506   681324505d5c        282MB
docker-dhcp-relay             latest                           a3dfa0aa88e9        292MB
docker-dhcp-relay             master.0-dirty-20200114.131506   a3dfa0aa88e9        292MB

Attach debug file sudo generate_dump:
sonic_dump_r-ethernet-sw230_20200130_153231.tar.gz

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions