[kdump] Fix OOM events in crashkernel

[Staphylo]

- Why I did it

A few issues where discovered with crashkernel on Arista platforms.

1. platforms using docker_inram=on would end up OOM in kdump environment.
   This happens because the same initramfs is used by SONiC and the crashkernel.
   With docker_inram=on the dockerfs.tar.gz is extracted in a tmpfs created for the occasion.
   Since dockerfs.tar.gz weights more than 1.5G, it doesn't fit into the kdump environment and ends up OOM.
   This OOM event can in turn trigger a panic.

2. Arista platforms with secureboot enabled would fail to load the crashkernel because the kernel parameter would be discarded on boot.
   This happens because the boot0 in secureboot mode is strict about kernel parameter injection.

3. The secureboot path allowlist would remove kernel crash reports.

4. The kdump service would fail on Arista products since /boot/ is empty in secureboot

- How I did it

1. To prevent an OOM event in the crashkernel the fix is to avoid the codepaths in union-mount that create tmpfs and populate them. Some more codepath specific to Arista devices are also skipped to make the kdump process faster.
   This relies on detecting that the initramfs is starting in a kdump environment and skipping some initialization.
   The /usr/sbin/kdump-config tool appends a few kernel cmdline arguments when loading the crashkernel.
   The most unique one is systemd.unit=kdump-tools.service which is used in a few initramfs hooks to set in_kdump.

2. To allow kdump to work in secureboot environment the cmdline generation in boot0 was slightly modified.
   The codepath to load kernel parameters changed by SONiC is now running for booting in secure mode.
   It was altered to prevent an append only behavior which would grow the kernel-cmdline at every reboot.
   This ever growing behavior would lead kexec to fail to load the kernel due to a too long cmdline.

3. To get the kernel crash under /var/crash this path has to be added to allowlist_paths

4. The /host/image-XXX/boot folder is now populated in secureboot mode but not used.

- How to verify it

Regular boot:

• enable kdump
• enable docker_inram=on via kernel-params
• reboot
• generate a crash echo c > /proc/sysrq-trigger
• before: witness OOM events on the console
• after: crash kernel works and crash available under /var/crash

Secure boot:

• enable kdump
• reboot
• generate a crash echo c > /proc/sysrq-trigger
• before: witness no kdump
• after: crash kernel works and crash available under /var/crash

- Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006
• 202012

This bug is present in 202012 and needs to be addressed.

- Description for the changelog

Fix kdump crashkernel errors seen on Arista devices

[Staphylo]

retest vsimage please

[Staphylo]

@jleveque @qiluo-msft

[qiluo-msft]

if $secureboot && $debug; then [](start = 8, length = 30)

if $secureboot && $debug; then [](start = 8, length = 30)

If secureboot and not debug, you will not use extra parameter. This is surprising different than the case of secureboot and debug. #Closed

[Staphylo]

That's correct.
The $target_path/$kernel_params file which is /host/kernel-params allows the user to inject more kernel parameters on the cmdline. It's especially useful during development to figure out which set of parameters should be used on a platform. But it can also be used to change some runtime behavior. Unfortunately some could have unfortunate behavior and end up compromising a system. To remove the risk of an attacker injecting harmful parameter we disable this feature in secureboot and print a message. It is still possible to boot the box in debug mode and inject parameters but that will show at runtime.
If you prefer I can disable this mechanism for secureboot entirely, including secureboot && debug.

[Staphylo]

retest vsimage please

[lguohan]

is this needed for 202012?

[Staphylo]

Yes it's needed for 202012