Support for updating tmpfs size according to Image size by lkunjumon · Pull Request #7484 · sonic-net/sonic-buildimage

lkunjumon · 2021-04-29T16:36:14Z

Why I did it

while sonic upgrade, Image will be extracted to tmpfs for installation so tmpfs size should be larger than image size. Image installation will fail if image size is larger than tmpfs size.

we are facing below error while installing debug image with size greater than tmpfs which is 1.5g in marvell armhf platform.

sonic-installer install
New image will be installed, continue? [y/N]: y
Downloading image...
...99%, 1744 MB, 708 KB/s, 0 seconds left...
Installing image SONiC-OS-202012.0-dirty-20210311.224845 and setting it as default...
Command: bash /tmp/sonic_image
tar: installer/fs.zip: Wrote only 7680 of 10240 bytes
tar: installer/onie-image-arm64.conf: Cannot write: No space left on device
tar: Exiting with failure status due to previous errors
Verifying image checksum ... OK.
Preparing image archive ...

How I did it

compare downloaded image size with tmpfs size, if size less than image size update the tmpfs size according to image size.

How to verify it

Install an Image with size larger than tmpfs. we verified by installing debug image with size 1.9gb which is larger than tmpfs size 1.5gb.

Which release branch to backport (provide reason below if selected)

201811
201911
202006
202012

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

qiluo-msft · 2021-04-29T19:55:25Z

installer/sharch_body.sh

 if [ "$(id -u)" = "0" ] ; then
    mount -t tmpfs tmpfs-installer $tmp_dir || exit 1
+    mount_size=$(df $tmp_dir | tail -1 | tr -s ' ' | cut -d' ' -f4)
+    if [ "$mount_size" -lt "$((image_size*3))" ]; then


image_size*3

This seems an estimation. How do you know *3 is enough? Could you get the accurate required size? #Closed

that must be generated from the build process.

use this to get the original size?

lgh@p330:~$ gzip -l docker-sonic-vs.gz compressed uncompressed ratio uncompressed_name 242618472 684168704 64.5% docker-sonic-vs

This script untars sonic-platform.bin, which is a ONIE image. We cannot use gzip command.
Will update the logic by getting image size from the "tar" command, excluding the ONIE installer script

Blueve · 2021-05-08T04:02:07Z

I hit same issue on Celestica platform.
The tmpfs is 991M by default but some of image size are larger than 1G

lkunjumon · 2021-05-17T13:03:54Z

Updated logic to get image size from "tar" command.

qiluo-msft · 2021-05-17T23:11:31Z

installer/sharch_body.sh


 echo " OK."

+image_size=$(( $(sed -e '1,/^exit_marker$/d' "$0"  | tar --to-stdout -xf - | wc -c) / 1024))


/

Do you want to get the ceiling of division?

okay.will update with ceiling.

Sorry, how do you get the ceiling? Also there are 2 more division below.

Instead of x / 1024,
(x + 1023) / 1024 will be much safer.

we can avoid ceiling here as we already adding ceiling to image_size by 1G in this command " mount_size=$(((image_size/1024/1024)+1)) " .Here if we add it will be ceiled by 1KB only ,so it is not necessary.

Okay, you are using padding instead of ceiling. Do you think 1G padding is too large?

"mount_size=$(((image_size/1024/1024)+1))" Here we are adding ceiling for image_size and extra free space for the mount , so extra free space will be less than 1GB.

My only concern is the extreme padding size 1G seem too large. Others look good to me.

@qiluo-msft,
is it fine to update 32MB extra space? Find below changes. If this is fine, we will push the changes.

mount_size=$((((image_size+1023)/1024) + 32))
mount -o remount,size="${mount_size}M" -t tmpfs tmpfs-installer $tmp_dir || exit 1

32M padding is good. However you divided 3 times, so I think you formula will not work for corner cases.
This is my math

bytes = $(sed -e '1,/^exit_marker$/d' "$0" | tar --to-stdout -xf - | wc -c) image_size=$((bytes + 1023) / 1024) mount_size=$((image_size + 31) / 32) * 32 mount -o remount,size="${mount_size}M"

32M padding is good. However you divided 3 times, so I think you formula will not work for corner cases.
This is my math

bytes = $(sed -e '1,/^exit_marker$/d' "$0" | tar --to-stdout -xf - | wc -c) image_size=$((bytes + 1023) / 1024) mount_size=$((image_size + 31) / 32) * 32 mount -o remount,size="${mount_size}M"

Updated PR with your comments. Please review.

qiluo-msft · 2021-05-17T23:12:18Z

installer/sharch_body.sh

 if [ "$(id -u)" = "0" ] ; then
    mount -t tmpfs tmpfs-installer $tmp_dir || exit 1
+    mount_size=$(df $tmp_dir | tail -1 | tr -s ' ' | cut -d' ' -f4)
+    if [ "$mount_size" -lt "$((image_size*3))" ]; then


image_size

Is image_sizes*1 is good enough? #Closed

I have addressed the comment. Please review

lkunjumon · 2021-05-28T09:47:40Z

retest please

radha-danda · 2021-06-09T15:20:05Z

@lguohan, @qiluo-msft, we have answered the queries on this PR. Kindly let us know if you have any further queries. Can this PR be merged?

Signed-off-by: lkunjumon <lkunjumon@marvell.com>

radha-danda · 2021-06-21T06:32:23Z

@lguohan, @qiluo-msft, we have answered the queries on this PR. Kindly let us know if you have any further queries. Can this PR be merged?

radha-danda · 2021-06-21T06:34:39Z

@lguohan, can this PR be merged as it is approved by Qi Luo

#### Why I did it while sonic upgrade, Image will be extracted to tmpfs for installation so tmpfs size should be larger than image size. Image installation will fail if image size is larger than tmpfs size. we are facing below error while installing debug image with size greater than tmpfs which is 1.5g in marvell armhf platform. sonic-installer install <url> New image will be installed, continue? [y/N]: y Downloading image... ...99%, 1744 MB, 708 KB/s, 0 seconds left... Installing image SONiC-OS-202012.0-dirty-20210311.224845 and setting it as default... Command: bash /tmp/sonic_image tar: installer/fs.zip: Wrote only 7680 of 10240 bytes tar: installer/onie-image-arm64.conf: Cannot write: No space left on device tar: Exiting with failure status due to previous errors Verifying image checksum ... OK. Preparing image archive ... #### How I did it compare downloaded image size with tmpfs size, if size less than image size update the tmpfs size according to image size. #### How to verify it Install an Image with size larger than tmpfs. we verified by installing debug image with size 1.9gb which is larger than tmpfs size 1.5gb.

dflynn-Nokia · 2021-09-03T13:41:46Z

Please cherry-pick this PR to the 202012 and 202106 releases.

#### Why I did it while sonic upgrade, Image will be extracted to tmpfs for installation so tmpfs size should be larger than image size. Image installation will fail if image size is larger than tmpfs size. we are facing below error while installing debug image with size greater than tmpfs which is 1.5g in marvell armhf platform. sonic-installer install <url> New image will be installed, continue? [y/N]: y Downloading image... ...99%, 1744 MB, 708 KB/s, 0 seconds left... Installing image SONiC-OS-202012.0-dirty-20210311.224845 and setting it as default... Command: bash /tmp/sonic_image tar: installer/fs.zip: Wrote only 7680 of 10240 bytes tar: installer/onie-image-arm64.conf: Cannot write: No space left on device tar: Exiting with failure status due to previous errors Verifying image checksum ... OK. Preparing image archive ... #### How I did it compare downloaded image size with tmpfs size, if size less than image size update the tmpfs size according to image size. #### How to verify it Install an Image with size larger than tmpfs. we verified by installing debug image with size 1.9gb which is larger than tmpfs size 1.5gb.

Support for updating tmpfs size according to Image size

33bc329

lkunjumon requested a review from qiluo-msft as a code owner April 29, 2021 16:36

qiluo-msft reviewed Apr 29, 2021

View reviewed changes

Add support to get actual size of compressed image.

25d32a1

qiluo-msft reviewed May 17, 2021

View reviewed changes

Updating tmpfs size according to Image size

3c6c399

Updated review comments and added extra 32MB free space for mount size.

53e8c97

Signed-off-by: lkunjumon <lkunjumon@marvell.com>

qiluo-msft approved these changes Jun 16, 2021

View reviewed changes

qiluo-msft merged commit 90801dc into sonic-net:master Jun 21, 2021

Blueve added Request for 202012 Branch Request for 202106 Branch labels Sep 9, 2021

qiluo-msft added the Included in 202012 Branch label Sep 13, 2021

judyjoseph added the Included in 202106 Branch label Sep 14, 2021


		echo " OK."

		image_size=$(( $(sed -e '1,/^exit_marker$/d' "$0" \| tar --to-stdout -xf - \| wc -c) / 1024))

Conversation

lkunjumon commented Apr 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

qiluo-msft Apr 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lguohan May 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antony-rheneus May 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Blueve commented May 8, 2021

Uh oh!

lkunjumon commented May 17, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lkunjumon Jun 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft May 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lkunjumon commented May 28, 2021

Uh oh!

radha-danda commented Jun 9, 2021

Uh oh!

radha-danda commented Jun 21, 2021

Uh oh!

radha-danda commented Jun 21, 2021

Uh oh!

dflynn-Nokia commented Sep 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

lkunjumon commented Apr 29, 2021 •

edited

Loading

qiluo-msft Apr 29, 2021 •

edited

Loading

lguohan May 8, 2021 •

edited

Loading

antony-rheneus May 17, 2021 •

edited

Loading

lkunjumon Jun 1, 2021 •

edited

Loading

qiluo-msft May 17, 2021 •

edited

Loading