Skip to content

[Sflow]: Kernel panic is seen when packets are received in sflow interface with different packet length #3991

Description

@chitra-raghavan

Description

[Sflow] : Kernel panic is seen upon sending Icmp packets with different packet size to interface in which sflow is enabled

        admin@vlab-01:~$ show sflowsFlow Global Information:
  sFlow Admin State:          up
  sFlow Polling Interval:     60
  sFlow AgentID:              default  1 Collectors configured:
    Name: collector_1         IP addr: 192.168.0.3UDP port: 6343
admin@vlab-01:~$ show sflow interfacesFlow interface configurations
+-------------+---------------+-----------------+
| Interface   | Admin State   |   Sampling Rate |
+=============+===============+=================+
| Ethernet112 | up            |             512 |

ping the DUT interface ip from peer continuously with different packet size. Check for kernel crash

admin@vlab-01:~$ timed out waiting for input: auto-logoutDebian GNU/Linux 9 vlab-01 ttyS0vlab-01 login: [16645.523275] skbuff: skb_over_panic: text:ffffffffc0577382 len:196 put:136 head:ffff90aae8d9a400 data:ffff90aae8d9a400 tail:0xc4 end:0xc0 dev:<NULL>
[16645.539991] ------------[ cut here ]------------
[16645.546260] kernel BUG at /sonic/src/sonic-linux-kernel/linux-4.9.168/net/core/skbuff.c:105!
[16645.552584] invalid opcode: 0000 [#1] SMP
[16645.554759] Modules linked in: cls_matchall sch_ingress tun 8021q garp mrp dummy team_mode_loadbalance team ip6table_mangle xt_TCPMSS xt_tcpudp iptable_mangle xt_conntrack nf_nat nf_conntrack_netlink nfnetlink nf_conntrack xfrm_user xfrm_algo joydev kvm_intel kvm irqbypass pcspkr evdev serio_raw bonding virtio_balloon shpchp button ebtable_broute bridge stp llc ebtable_nat ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter act_sample psample ip_tables x_tables autofs4 loop ext4 crc16 jbd2 crc32c_generic fscrypto ecb glue_helper lrw gf128mul ablk_helper cryptd aes_x86_64 mbcache nls_utf8 nls_cp437 nls_ascii vfat fat overlay squashfs virtio_blk psmouse ata_generic ata_piix virtio_pci uhci_hcd virtio_ring virtio ehci_hcd libata floppy e1000 i2c_piix4 scsi_mod usbcore usb_common
[16645.599686] CPU: 1 PID: 16 Comm: ksoftirqd/1 Not tainted 4.9.0-9-2-amd64 #1 Debian 4.9.168-1+deb9u5
[16645.604164] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[16645.607062] task: ffff90ab38240240 task.stack: ffffaa430039c000
[16645.609611] RIP: 0010:[<ffffffffbb6faca0>]  [<ffffffffbb6faca0>] skb_panic+0x60/0x70
[16645.614309] RSP: 0018:ffffaa430039fbd8  EFLAGS: 00010286
[16645.617380] RAX: 0000000000000087 RBX: 0000000000000082 RCX: 0000000000000000
[16645.621205] RDX: 0000000000000000 RSI: ffff90ab3cd10648 RDI: ffff90ab3cd10648
[16645.624596] RBP: ffff90aae5101c00 R08: 0000000000000000 R09: 00000000bc0ba601
[16645.628025] R10: 0000000000000014 R11: ffffffffbc0ba5ad R12: ffffaa430039fc1c
[16645.631757] R13: ffff90aae790b400 R14: 0000000000000000 R15: ffff90aae74e6940
[16645.635100] FS:  0000000000000000(0000) GS:ffff90ab3cd00000(0000) knlGS:0000000000000000
[16645.638971] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[16645.642169] CR2: 00007fb73002d118 CR3: 000000003b0d8000 CR4: 0000000000000670
[16645.646171] Stack:
[16645.647368]  ffff90aae8d9a400 00000000000000c4 00000000000000c0 ffffffffbbc15029
[16645.651822]  ffffffffbb6fb279 ffffffffc0577382 00000200e5101c00 ffff90aae8d9a414
[16645.656294]  00000095bbedc040 4e4a7bd24042a724 ffff90aae53b6a00 ffff90aae790b400
[16645.660722] Call Trace:
[16645.662107]  [<ffffffffbb6fb279>] ? skb_put+0x49/0x50
[16645.665032]  [<ffffffffc0577382>] ? psample_sample_packet+0x182/0x2d0 [psample]
[16645.669089]  [<ffffffffc059657f>] ? tcf_sample_act+0xbf/0x1a0 [act_sample]
[16645.672564]  [<ffffffffbb74255c>] ? tcf_action_exec+0x4c/0x80
[16645.674975]  [<ffffffffbb73de52>] ? tc_classify+0x72/0x100
[16645.678160]  [<ffffffffbb710f51>] ? __netif_receive_skb_core+0x461/0xa40
[16645.682964]  [<ffffffffbb81b7b0>] ? __switch_to_asm+0x40/0x70
[16645.686507]  [<ffffffffbb81b7a4>] ? __switch_to_asm+0x34/0x70
[16645.690525]  [<ffffffffbb81b7b0>] ? __switch_to_asm+0x40/0x70
[16645.694761]  [<ffffffffbb81b7a4>] ? __switch_to_asm+0x34/0x70
[16645.698873]  [<ffffffffbb712668>] ? process_backlog+0x88/0x130
[16645.702652]  [<ffffffffbb711df6>] ? net_rx_action+0x246/0x380
[16645.705900]  [<ffffffffbb81f36d>] ? __do_softirq+0x10d/0x2b0
[16645.709678]  [<ffffffffbb29d560>] ? sort_range+0x20/0x20
[16645.712121]  [<ffffffffbb27ff5e>] ? run_ksoftirqd+0x1e/0x40
[16645.715454]  [<ffffffffbb29d66e>] ? smpboot_thread_fn+0x10e/0x160
[16645.719348]  [<ffffffffbb299dd9>] ? kthread+0xd9/0xf0
[16645.722980]  [<ffffffffbb299d00>] ? kthread_park+0x60/0x60
[16645.727181]  [<ffffffffbb81b837>] ? ret_from_fork+0x57/0x70
[16645.730174] Code: cc 00 00 00 48 89 44 24 10 8b 87 c8 00 00 00 48 89 44 24 08 48 8b 87 d8 00 00 00 48 c7 c7 88 3c c6 bb 48 89 04 24 e8 6c 5f c8 ff <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 e9 7b 70 e4 ff
[16645.748125] RIP  [<ffffffffbb6faca0>] skb_panic+0x60/0x70
[16645.751091]  RSP <ffffaa430039fbd8>
[16645.754225] ---[ end trace e5348bf22a10a918 ]---
[16645.756500] Kernel panic - not syncing: Fatal exception in interrupt
[16645.760491] Kernel Offset: 0x3a200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[16645.765845] Rebooting in 10 seconds..

Steps to reproduce the issue:

  1. configure ip address on an interface
  2. enable sflow on that interface
  3. ping the interface IP address with different packet size from peer.
  4. Check for crash

Describe the results you received:
Kenel panic is seen

Describe the results you expected:

Sflow should the sample packets and send to collector

Additional information you deem important (e.g. issue happens only occasionally):
TESTED in SONIC-VS

**Output of `show version`:**
    root@vlab-01:~# show vers

SONiC Software Version: SONiC.HEAD.188-4458efbd
Distribution: Debian 9.11
Kernel: 4.9.0-9-2-amd64
Build commit: 4458efbd
Build date: Wed Dec 18 08:42:30 UTC 2019
Built by: johnar@jenkins-worker-11

Platform: x86_64-kvm_x86_64-r0
HwSKU: Force10-S6000
ASIC: vs
Serial Number: 000000
Uptime: 10:03:26 up 58 min,  1 user,  load average: 0.41, 0.27, 0.30

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-syncd-vs            HEAD.188-4458efbd   495cc11160dd        291MB
docker-syncd-vs            latest              495cc11160dd        291MB
docker-platform-monitor    HEAD.188-4458efbd   07626700bce9        333MB
docker-platform-monitor    latest              07626700bce9        333MB
docker-fpm-frr             HEAD.188-4458efbd   5b99dc1e2a92        325MB
docker-fpm-frr             latest              5b99dc1e2a92        325MB
docker-lldp-sv2            HEAD.188-4458efbd   f40b3e9c0a24        303MB
docker-lldp-sv2            latest              f40b3e9c0a24        303MB
docker-dhcp-relay          HEAD.188-4458efbd   f56f85f26935        290MB
docker-dhcp-relay          latest              f56f85f26935        290MB
docker-database            HEAD.188-4458efbd   452bb9eae044        282MB
docker-database            latest              452bb9eae044        282MB
docker-sflow               HEAD.188-4458efbd   a7b040a3c4f1        305MB
docker-sflow               latest              a7b040a3c4f1        305MB
docker-teamd               HEAD.188-4458efbd   05a8b6083599        305MB
docker-teamd               latest              05a8b6083599        305MB
docker-snmp-sv2            HEAD.188-4458efbd   35d2e24c3598        339MB
docker-snmp-sv2            latest              35d2e24c3598        339MB
docker-orchagent           HEAD.188-4458efbd   71397b40a481        323MB
docker-orchagent           latest              71397b40a481        323MB
docker-sonic-telemetry     HEAD.188-4458efbd   8cca2b75a13d        305MB
docker-sonic-telemetry     latest              8cca2b75a13d        305MB
docker-router-advertiser   HEAD.188-4458efbd   820d05e70b3c        282MB
docker-router-advertiser   latest              820d05e70b3c        282MB

root@vlab-01:~#
**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions