sonic-platform-modules-cel dx010: speed up dx010 platform init script by qnos · Pull Request #10237 · sonic-net/sonic-buildimage

qnos · 2022-03-15T01:23:45Z

Remove installing of python2 sonic platform wheel for dx010
Optimize dx010 sonic platform init script to speed up init process
Fix issue [warm-upgrade][202012] Slow Celestica platform init in rc.local causes lacp-teardown #10152: [warm-upgrade][202012] Slow Celestica platform init
in rc.local causes lacp-teardown

Signed-off-by: Eric Zhu [email protected]

Why I did it

To fix issue #10152 for dx010.
202012 Warm upgrade causes lacp-teardown on Dx010 TOR. platform code initialize slow causing lacp timeout.

How I did it

Remove the python2 sonic platform wheel which is deprecated.
Optimize the dx010 sonic platform script to speed up the init process.

How to verify it

Check the warm reboot log, warm reboot is 8-9s faster than before.

[945232.436622] kexec_core: Starting new kernel                                                                                                                                                          
[    5.838762] rc.local[461]: + sed -e s/build_version: //g;s/'//g                                                                                                                                       
[    5.863209] rc.local[460]: + grep build_version                                                                                                                                                       
[    5.870373] rc.local[459]: + cat /etc/sonic/sonic_version.yml          
[    5.882392] rc.local[444]: + SONIC_VERSION=CLS-202012-f93d1f64a2_220311_0001                                                                                                                          
[    5.900265] rc.local[444]: + FIRST_BOOT_FILE=/host/image-CLS-202012-f93d1f64a2_220311_0001/platform/firsttime                                                                                         
[    5.919713] rc.local[444]: + SONIC_CONFIG_DIR=/host/image-CLS-202012-f93d1f64a2_220311_0001/sonic-config                                                                                              
[    5.939963] rc.local[444]: + SONIC_ENV_FILE=/host/image-CLS-202012-f93d1f64a2_220311_0001/sonic-config/sonic-environment                                                                              
[    5.959775] rc.local[444]: + [ -d /host/image-CLS-202012-f93d1f64a2_220311_0001/sonic-config -a -f /host/image-CLS-202012-f93d1f64a2_220311_0001/sonic-config/sonic-environment ]                     
[    5.991680] rc.local[444]: + echo moving file /host/image-CLS-202012-f93d1f64a2_220311_0001/sonic-config/sonic-environment to /etc/sonic                                                              
[    6.015772] rc.local[444]: moving file /host/image-CLS-202012-f93d1f64a2_220311_0001/sonic-config/sonic-environment to /etc/sonic                                                                     
[    6.039698] rc.local[444]: + mv /host/image-CLS-202012-f93d1f64a2_220311_0001/sonic-config/sonic-environment /etc/sonic                                                                               
[    6.063968] rc.local[444]: + logger SONiC version CLS-202012-f93d1f64a2_220311_0001 starting up...                                                                                                    
[    6.075689] rc.local[444]: + grub_installation_needed=                                                                                                                                                
[    6.084995] rc.local[444]: + [ ! -e /host/machine.conf ]                                                                                                                                              
[    6.099820] rc.local[444]: + migrate_nos_configuration                                                                                                                                                
[    6.115707] rc.local[444]: + rm -rf /host/migration                                                                                                                                                   
[    6.136062] rc.local[444]: + mkdir -p /host/migration                                                                                                                                                 
[    6.154299] kdump-tools[428]: Starting kdump-tools:                                                                                                                                                   
[    6.169527] rc.local[502]: + cat /proc/cmdline                                                                                                                                                        
[    6.175936] kdump-tools[465]: no crashkernel= parameter in the kernel cmdline ...                                                                                                                     
[    6.199530] kdump-tools[512]:  failed!                                                                                                                                                                
[    6.214259] rc.local[444]: + set -- BOOT_IMAGE=/image-CLS-202012-f93d1f64a2_220311_0001/boot/vmlinuz-4.19.0-12-2-amd64 root=UUID=9695eb38-b94c-46d6-ada1-c2a790617c0e rw console=tty0 console=ttyS0,11
5200n8 quiet intel_idle.max_cstate=0 net.ifnames=0 biosdevname=0 loop=image-CLS-202012-f93d1f64a2_220311_0001/fs.squashfs loopfstype=squashfs systemd.unified_cgroup_hierarchy=0 apparmor=1 security=appa
rmor varlog_size=4096 usbcore.autosuspend=-1 module_blacklist=gpio_ich SONIC_BOOT_TYPE=warm                                                                                                              
[    6.271805] rc.local[444]: + [ -n  ]                                                                                                                                                                  
[    6.283662] rc.local[444]: + . /host/machine.conf                                            
[    6.299681] rc.local[444]: + onie_version=2014.08.0.0.6                                                                                                                                               
[    6.315716] rc.local[444]: + onie_vendor_id=12244                                                                                                                                                     
[    6.332318] rc.local[444]: + onie_platform=x86_64-cel_seastone-r0                                                                                                                                     
[    6.349204] rc.local[444]: + onie_machine=cel_seastone                                                                                                                                                
[    6.363709] rc.local[444]: + onie_machine_rev=0                                                                                                                                                       
[    6.379679] rc.local[444]: + onie_arch=x86_64                                                                                                                                                         
[    6.400461] rc.local[444]: + onie_config_version=1                                                                                                                                                    
[    6.415694] rc.local[444]: + onie_build_date=2016-07-19T12:10-0400                                                                                                                                    
[    6.431697] rc.local[444]: + onie_partition_type=gpt                                                                                                                                                  
[    6.447759] rc.local[444]: + onie_kernel_version=3.2.35                                                                                                                                               
[    6.463741] rc.local[444]: + program_console_speed                                                                                                                                                    
[    6.472723] rc.local[505]: + grep -Eo console=ttyS[0-9]+,[0-9]+                                                                                                                                       
[    6.492563] rc.local[504]: + cat /proc/cmdline                                                                                                                                                        
[    6.508658] rc.local[506]: + cut -d , -f2                                                                                                                                                             
[    6.526088] rc.local[444]: + speed=115200                                                                                                                                                             
[    6.543885] rc.local[444]: + [ -z 115200 ]                                                                                                                                                            
[    6.555790] rc.local[444]: + CONSOLE_SPEED=115200                                                                                                                                                     
[    6.562351] rc.local[507]: + grep agetty /lib/systemd/system/[email protected]                                                                                                                    
[    6.585391] rc.local[508]: + grep keep-baud                                                                                                                                                           
[    6.603647] rc.local[508]: ExecStart=-/sbin/agetty -o '-p -- \\u' --keep-baud 115200,57600,38400,9600 %I $TERM                                                                                        
[    6.624179] rc.local[444]: + [ 0 = 0 ]                                                                                                                                                                
[    6.639718] rc.local[444]: + sed -i s|\-\-keep\-baud .* %I| 115200 %I|g /lib/systemd/system/[email protected]                                                                                     
[    6.659655] rc.local[444]: + systemctl daemon-reload                                                                                                                                                  
[    6.675673] rc.local[444]: + [ -f /host/image-CLS-202012-f93d1f64a2_220311_0001/platform/firsttime ]      
[    6.699699] rc.local[444]: + echo First boot detected. Performing first boot tasks...                                                                                                                 
[    6.719684] rc.local[444]: First boot detected. Performing first boot tasks...                                                                                                                        
[    6.739674] rc.local[444]: + [ -n  ]                                                                                                                                                                  
[    6.751663] rc.local[444]: + [ -n x86_64-cel_seastone-r0 ]                                                                                                                                            
[    6.767624] rc.local[444]: + platform=x86_64-cel_seastone-r0                                                                                                                                          
[    6.783625] rc.local[444]: + [ -d /host/old_config ]                                                                                                                                                  
[    6.799618] rc.local[444]: + mv -f /host/old_config /etc/sonic/                                                                                                                                       
[    6.816864] rc.local[444]: + rm -rf /etc/sonic/old_config/old_config                                                                                                                                  
[    6.835670] rc.local[444]: + touch /tmp/pending_config_migration                                                                                                                                      
[    6.851630] rc.local[444]: + touch /tmp/notify_firstboot_to_platform                                                                                                                                  
[    6.867609] rc.local[444]: + [ ! -d /host/reboot-cause/platform ]                                                                                                                                     
[    6.884516] rc.local[444]: + [ -d /host/image-CLS-202012-f93d1f64a2_220311_0001/platform/x86_64-cel_seastone-r0 ]                                                                                     
[    6.907732] rc.local[444]: + dpkg -i /host/image-CLS-202012-f93d1f64a2_220311_0001/platform/x86_64-cel_seastone-r0/platform-modules-dx010_0.9_amd64.deb                                               
[    6.933086] rc.local[551]: Selecting previously unselected package platform-modules-dx010.                                                                                                            
[    7.518234] rc.local[551]: (Reading database ... 30055 files and directories currently installed.)                                                                                                    
[    7.539760] rc.local[551]: Preparing to unpack .../platform-modules-dx010_0.9_amd64.deb ...                                                                                                           
[    7.563705] rc.local[551]: Unpacking platform-modules-dx010 (0.9) ...                                                                                                                                 
[    8.090707] rc.local[551]: Setting up platform-modules-dx010 (0.9) ...                                                                                                                                
[   12.777431] rc.local[611]: Synchronizing state of platform-modules-dx010.service with SysV service script with /lib/systemd/systemd-sysv-install.                                                     
[   12.807716] rc.local[611]: Executing: /lib/systemd/systemd-sysv-install enable platform-modules-dx010                                                                                                 
[   23.526475] rc.local[993]: Processing /usr/share/sonic/device/x86_64-cel_seastone-r0/sonic_platform-1.0-py3-none-any.whl                                                                              
[   24.213094] rc.local[993]: Installing collected packages: sonic-platform                                                                                                                              
[   24.376318] rc.local[993]: Successfully installed sonic-platform-1.0 
[   24.391815] rc.local[993]: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv                                                                                                                                                       │
[   26.458719] rc.local[444]: + sync                                                                                                                                                                     
[   27.503835] rc.local[444]: + [ -n x86_64-cel_seastone-r0 ]                                                                                                                                            
[   27.519767] rc.local[444]: + [ -n  ]                                                                                                                                                                  
[   27.531894] rc.local[444]: + mkdir -p /var/platform                                                                                                                                                   
[   27.547755] rc.local[444]: + ebtables_config                                                                                                                                                          
[   27.563785] rc.local[444]: + /usr/sbin/ebtables-restore                                                                                                                                               
[   27.579849] rc.local[444]: + /usr/sbin/ebtables -t filter --atomic-file /etc/ebtables.filter --atomic-save                                                                                            
[   27.599756] rc.local[444]: + sed -i -e s/__PLATFORM__/x86_64-cel_seastone-r0/g /etc/default/kdump-tools                                                                                               
[   27.619761] rc.local[444]: + firsttime_exit                                                                                                                                                           
[   27.635769] rc.local[444]: + rm -rf /host/image-CLS-202012-f93d1f64a2_220311_0001/platform/firsttime                                                                                                  
[   27.655752] rc.local[444]: + exit 0

Setup up Port-channel between two DUT, warm upgrade one DUT, and do not see LAG flap in syslog.

Which release branch to backport (provide reason below if selected)

No need to backport.

* Remove installing of python2 sonic platform wheel for dx010 * Optimize dx010 sonic platform init script to speed up init process * Fix issue #10152: [warm-upgrade][202012] Slow Celestica platform init in rc.local causes lacp-teardown Signed-off-by: Eric Zhu <[email protected]>

ghost · 2022-03-15T01:23:58Z

All CLA requirements met.

qnos · 2022-03-15T10:51:35Z

Why compile failed

Build failure was caused by libexpat1-dev and libexpat1 package dependency relationship mismatch. So the libexpat1-dev could not been installed as libexpat1 2.2.6-2+deb10u4 already installed, which is a dependency for python-dev. The libexpat1 2.2.6-2+deb10u4 is included in http://packages.trafficmanager.net/debian/debian-security repo added in files/apt/sources.list.amd64.
The repo was introduced in the following commit:
fa7d636

When install apparmor in build_debian.sh, the libexpat1 2.2.6-2+deb10u4 was installed as a dependency, which caused dependency mismatch.

How to solve it

Keep deb packages in http://packages.trafficmanager.net/debian/ and http://debian-archive.trafficmanager.net/ version consistent.
Workaround:

Method 1: Temperately remove conflict apt repo from files/apt/sources.list.amd64.**

$ git diff files/apt/sources.list.amd64
diff --git a/files/apt/sources.list.amd64 b/files/apt/sources.list.amd64
index 45902be10..c38e9a57e 100644
--- a/files/apt/sources.list.amd64
+++ b/files/apt/sources.list.amd64
@@ -8,6 +8,6 @@ deb-src [arch=amd64] http://debian-archive.trafficmanager.net/debian-security/ b
 deb [arch=amd64] http://debian-archive.trafficmanager.net/debian/ buster-backports main contrib non-free
 deb [arch=amd64] http://packages.trafficmanager.net/debian/debian buster main contrib non-free
 deb-src [arch=amd64] http://packages.trafficmanager.net/debian/debian buster main contrib non-free
-deb [arch=amd64] http://packages.trafficmanager.net/debian/debian-security/ buster_updates main contrib non-free
+#deb [arch=amd64] http://packages.trafficmanager.net/debian/debian-security/ buster_updates main contrib non-free
 deb-src [arch=amd64] http://packages.trafficmanager.net/debian/debian-security/ buster_updates main contrib non-free
 deb [arch=amd64] http://packages.trafficmanager.net/debian/debian buster-backports main contrib non-free

Method 2: Add following into files/build_templates/sonic_debian_extension.j2 to downgrade libexpat1 before install python-dev and python3-dev.**

sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y --allow-downgrades install \
	libexpat1=2.2.6-2+deb10u3

Debug log

The following packages have unmet dependencies:
 libexpat1-dev : Depends: libexpat1 (= 2.2.6-2+deb10u3) but 2.2.6-2+deb10u4 is to be installed
E: Unable to correct problems, you have held broken packages.


root@dd47c6e7198e:/# apt search libexpat1
Sorting... Done
Full Text Search... Done
libexpat1/now **2.2.6-2+deb10u4** amd64 [installed,local]
  XML parsing C library - runtime library

libexpat1-dev/oldstable **2.2.6-2+deb10u3** amd64
  XML parsing C library - development kit

root@dd47c6e7198e:/#

+ sudo LANG=C chroot ./fsroot apt-get -y install apparmor             
Reading package lists...                                              
Building dependency tree...                                           
Reading state information...                                          
The following additional packages will be installed:                  
  libexpat1 libmpdec2 libpython3-stdlib libpython3.7-minimal          
  libpython3.7-stdlib libreadline7 libsqlite3-0 mime-support python3  
  python3-minimal python3.7 python3.7-minimal readline-common         
Suggested packages: 

# libexpat1 in packages.trafficmanager.net repo
$ sudo apt download libexpat1
Get:1 http://packages.trafficmanager.net/debian/debian-security buster_updates/main amd64 libexpat1 amd64 2.2.6-2+deb10u4 [108 kB]

lguohan · 2022-03-21T00:24:52Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2022-03-21T00:25:03Z

Azure Pipelines successfully started running 1 pipeline(s).

lguohan · 2022-03-21T00:34:31Z

can you make sure we have such fix in master branch as well?

qnos · 2022-03-21T01:02:07Z

can you make sure we have such fix in master branch as well?

I only requested to merge into 202012 branch, but it would be better to also merge into master to solve the issue for future release.

vaibhavhd · 2022-03-22T01:33:49Z

can you make sure we have such fix in master branch as well?

I only requested to merge into 202012 branch, but it would be better to also merge into master to solve the issue for future release.

Please raise a PR for master branch as well. Ideally if the fix is needed on both branches, we first raise a master PR, and then cherrypick the change into 202012.

qnos · 2022-03-22T02:28:19Z

can you make sure we have such fix in master branch as well?

I only requested to merge into 202012 branch, but it would be better to also merge into master to solve the issue for future release.

Please raise a PR for master branch as well. Ideally if the fix is needed on both branches, we first raise a master PR, and then cherrypick the change into 202012.

Thanks. New PR #10313 is raised to merge the fix into master branch. PR #10237 has committed the fix into branch 202012.

qnos mentioned this pull request Mar 15, 2022

[warm-upgrade][202012] Slow Celestica platform init in rc.local causes lacp-teardown #10152

Closed

vaibhavhd requested a review from sujinmkang March 17, 2022 00:24

sujinmkang approved these changes Mar 21, 2022

View reviewed changes

sujinmkang merged commit deac4ee into sonic-net:202012 Mar 21, 2022

qnos mentioned this pull request Mar 22, 2022

sonic-platform-modules-cel dx010: speed up dx010 platform init script #10313

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sonic-platform-modules-cel dx010: speed up dx010 platform init script#10237

sonic-platform-modules-cel dx010: speed up dx010 platform init script#10237
sujinmkang merged 1 commit intosonic-net:202012from
qnos:202012

qnos commented Mar 15, 2022 •

edited

Loading

Uh oh!

ghost commented Mar 15, 2022 •

edited by ghost

Loading

Uh oh!

qnos commented Mar 15, 2022 •

edited

Loading

Uh oh!

lguohan commented Mar 21, 2022

Uh oh!

azure-pipelines bot commented Mar 21, 2022

Uh oh!

lguohan commented Mar 21, 2022

Uh oh!

qnos commented Mar 21, 2022

Uh oh!

vaibhavhd commented Mar 22, 2022

Uh oh!

qnos commented Mar 22, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

qnos commented Mar 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Uh oh!

ghost commented Mar 15, 2022 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qnos commented Mar 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why compile failed

How to solve it

Debug log

Uh oh!

lguohan commented Mar 21, 2022

Uh oh!

azure-pipelines bot commented Mar 21, 2022

Uh oh!

lguohan commented Mar 21, 2022

Uh oh!

qnos commented Mar 21, 2022

Uh oh!

vaibhavhd commented Mar 22, 2022

Uh oh!

qnos commented Mar 22, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

qnos commented Mar 15, 2022 •

edited

Loading

ghost commented Mar 15, 2022 •

edited by ghost

Loading

qnos commented Mar 15, 2022 •

edited

Loading