Support SONiC Reproduceable Build-debian/pip/web packages #5718
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -40,6 +40,8 @@ PLATFORM_DIR=platform | |
| ## Hostname for the linux image | ||
| HOSTNAME=sonic | ||
| DEFAULT_USERINFO="Default admin user,,," | ||
| BUILD_TOOL_PATH=src/sonic-build-hooks/buildinfo | ||
| TRUSTED_GPG_DIR=$BUILD_TOOL_PATH/trusted.gpg.d | ||
|
|
||
| ## Read ONIE image related config file | ||
| . ./onie-image.conf | ||
|
|
@@ -70,16 +72,12 @@ pushd $FILESYSTEM_ROOT | |
| sudo mount --bind . . | ||
| popd | ||
|
|
||
| ## Build the host debian base system | ||
| echo '[INFO] Build host debian base system...' | ||
| TARGET_PATH=$TARGET_PATH scripts/build_debian_base_system.sh $CONFIGURED_ARCH $IMAGE_DISTRO $FILESYSTEM_ROOT | ||
|
|
||
| # Prepare buildinfo | ||
| sudo scripts/prepare_debian_image_buildinfo.sh $CONFIGURED_ARCH $IMAGE_DISTRO $FILESYSTEM_ROOT $http_proxy | ||
|
|
||
xumia marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ## Config hostname and hosts, otherwise 'sudo ...' will complain 'sudo: unable to resolve host ...' | ||
| sudo LANG=C chroot $FILESYSTEM_ROOT /bin/bash -c "echo '$HOSTNAME' > /etc/hostname" | ||
|
|
@@ -100,6 +98,9 @@ echo '[INFO] Mount all' | |
| ## Output all the mounted device for troubleshooting | ||
| sudo LANG=C chroot $FILESYSTEM_ROOT mount | ||
|
|
||
| ## Install the trusted gpg public keys | ||
| [ -d $TRUSTED_GPG_DIR ] && [ ! -z "$(ls $TRUSTED_GPG_DIR)" ] && sudo cp $TRUSTED_GPG_DIR/* ${FILESYSTEM_ROOT}/etc/apt/trusted.gpg.d/ | ||
|
Collaborator
There was a problem hiding this comment. Is it better to install gpg file by
Collaborator
Author
There was a problem hiding this comment. See another comment in scrpts/prepare_docker_buildinfo.sh In reply to: 539809352 [](ancestors = 539809352) |
||
|
|
||
| ## Pointing apt to public apt mirrors and getting latest packages, needed for latest security updates | ||
| sudo cp files/apt/sources.list.$CONFIGURED_ARCH $FILESYSTEM_ROOT/etc/apt/sources.list | ||
| sudo cp files/apt/apt.conf.d/{81norecommends,apt-{clean,gzip-indexes,no-languages},no-check-valid-until} $FILESYSTEM_ROOT/etc/apt/apt.conf.d/ | ||
|
|
@@ -571,6 +572,8 @@ sudo du -hsx $FILESYSTEM_ROOT | |
| sudo mkdir -p $FILESYSTEM_ROOT/var/lib/docker | ||
| sudo mksquashfs $FILESYSTEM_ROOT $FILESYSTEM_SQUASHFS -e boot -e var/lib/docker -e $PLATFORM_DIR | ||
|
|
||
| scripts/collect_host_image_version_files.sh $TARGET_PATH $FILESYSTEM_ROOT | ||
|
|
||
| ## Compress docker files | ||
| pushd $FILESYSTEM_ROOT && sudo tar czf $OLDPWD/$FILESYSTEM_DOCKERFS -C ${DOCKERFS_PATH}var/lib/docker .; popd | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| #!/bin/bash | ||
|
|
||
| CONFIGURED_ARCH=$1 | ||
| IMAGE_DISTRO=$2 | ||
| FILESYSTEM_ROOT=$3 | ||
| http_proxy=$4 | ||
|
|
||
| TARGET=$TARGET_PATH | ||
| [ -z "$TARGET" ] && TARGET=target | ||
|
|
||
| . /usr/local/share/buildinfo/scripts/buildinfo_base.sh | ||
|
|
||
| TARGET_BASEIMAGE_PATH=$TARGET/versions/host-base-image | ||
| mkdir -p $TARGET_BASEIMAGE_PATH | ||
|
|
||
| generate_version_file() | ||
| { | ||
| sudo LANG=C chroot $FILESYSTEM_ROOT /bin/bash -c "dpkg-query -W -f '\${Package}==\${Version}\n'" > $TARGET_BASEIMAGE_PATH/versions-deb-${IMAGE_DISTRO}-${CONFIGURED_ARCH} | ||
| } | ||
|
|
||
| if [ "$ENABLE_VERSION_CONTROL_DEB" != "y" ]; then | ||
| if [[ $CONFIGURED_ARCH == armhf || $CONFIGURED_ARCH == arm64 ]]; then | ||
| # qemu arm bin executable for cross-building | ||
| sudo mkdir -p $FILESYSTEM_ROOT/usr/bin | ||
| sudo cp /usr/bin/qemu*static $FILESYSTEM_ROOT/usr/bin || true | ||
| sudo http_proxy=$HTTP_PROXY SKIP_BUILD_HOOK=y debootstrap --variant=minbase --arch $CONFIGURED_ARCH $IMAGE_DISTRO $FILESYSTEM_ROOT http://deb.debian.org/debian | ||
| else | ||
| sudo http_proxy=$HTTP_PROXY SKIP_BUILD_HOOK=y debootstrap --variant=minbase --arch $CONFIGURED_ARCH $IMAGE_DISTRO $FILESYSTEM_ROOT http://debian-archive.trafficmanager.net/debian | ||
| fi | ||
| RET=$? | ||
| if [ $RET -ne 0 ]; then | ||
| exit $RET | ||
| fi | ||
|
|
||
| generate_version_file | ||
| exit $RET | ||
| fi | ||
|
|
||
| ARCH=$(dpkg --print-architecture) | ||
| DISTRO=$(grep CODENAME /etc/os-release | cut -d= -f2) | ||
| if [ "$ARCH" != "$CONFIGURED_ARCH" ] || [ "$DISTRO" != "$IMAGE_DISTRO" ]; then | ||
| "Not support to build different ARCH/DISTRO ${CONFIGURED_ARCH}:${$IMAGE_DISTRO} in ${ARCH}:${DISTRO}." | ||
| exit 1 | ||
| fi | ||
|
|
||
| BASE_VERSIONS=files/build/versions/host-base-image/versions-deb-${IMAGE_DISTRO} | ||
| BASEIMAGE_TARBALLPATH=$TARGET/baseimage | ||
| BASEIMAGE_TARBALL=$(realpath -e $TARGET)/baseimage.tgz | ||
|
|
||
| rm -rf $BASEIMAGE_TARBALLPATH $BASEIMAGE_TARBALL | ||
|
|
||
| ARCHIEVES=$BASEIMAGE_TARBALLPATH/var/cache/apt/archives | ||
| APTLIST=$BASEIMAGE_TARBALLPATH/var/lib/apt/lists | ||
| TARGET_DEBOOTSTRAP=$BASEIMAGE_TARBALLPATH/debootstrap | ||
| APTDEBIAN="$APTLIST/deb.debian.org_debian_dists_buster_main_binary-${CONFIGURED_ARCH}_Packages" | ||
| DEBPATHS=$TARGET_DEBOOTSTRAP/debpaths | ||
| DEBOOTSTRAP_BASE=$TARGET_DEBOOTSTRAP/base | ||
| DEBOOTSTRAP_REQUIRED=$TARGET_DEBOOTSTRAP/required | ||
| [ -d $BASEIMAGE_TARBALLPATH ] && rm -rf $BASEIMAGE_TARBALLPATH | ||
| mkdir -p $ARCHIEVES | ||
| mkdir -p $APTLIST | ||
| mkdir -p $TARGET_DEBOOTSTRAP | ||
| PACKAGES=$(sed -E 's/=(=[^=]*)$/\1/' $BASE_VERSIONS) | ||
| URL_ARR=($(apt-get download --print-uris $PACKAGES | cut -d" " -f1 | tr -d "'")) | ||
| PACKAGE_ARR=($PACKAGES) | ||
| LENGTH=${#PACKAGE_ARR[@]} | ||
| for ((i=0;i<LENGTH;i++)) | ||
| do | ||
| package=${PACKAGE_ARR[$i]} | ||
| packagename=$(echo $package | sed -E 's/=[^=]*$//') | ||
| url=${URL_ARR[$i]} | ||
| filename=$(basename "$url") | ||
| SKIP_BUILD_HOOK=y wget $url -P $ARCHIEVES | ||
| echo $packagename >> $DEBOOTSTRAP_REQUIRED | ||
| echo "$packagename /var/cache/apt/archives/$filename" >> $DEBPATHS | ||
| done | ||
| touch $APTDEBIAN | ||
| touch $DEBOOTSTRAP_BASE | ||
| (cd $BASEIMAGE_TARBALLPATH && tar -zcf $BASEIMAGE_TARBALL .) | ||
|
|
||
| sudo debootstrap --verbose --variant=minbase --arch $CONFIGURED_ARCH --unpack-tarball=$BASEIMAGE_TARBALL $IMAGE_DISTRO $FILESYSTEM_ROOT | ||
| RET=$? | ||
| if [ $RET -ne 0 ]; then | ||
| exit $RET | ||
| fi | ||
|
|
||
| generate_version_file |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| #!/bin/bash | ||
|
|
||
| RET=$1 | ||
| BLDENV=$2 | ||
| TARGET_PATH=$3 | ||
|
|
||
| TIMESTAMP=$(date +"%Y%m%d%H%M%S") | ||
| . /usr/local/share/buildinfo/scripts/buildinfo_base.sh | ||
|
|
||
| [ -z "$BLDENV" ] && BLDENV=$(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) | ||
| [ -z "$BLDENV" ] && exit $RET | ||
|
|
||
| [ -z "$TARGET_PATH" ] && TARGET_PATH=./target | ||
|
|
||
| VERSION_BUILD_PATH=$TARGET_PATH/versions/build | ||
| VERSION_SLAVE_PATH=$VERSION_BUILD_PATH/build-sonic-slave-${BLDENV} | ||
| LOG_VERSION_PATH=$VERSION_BUILD_PATH/log-${TIMESTAMP} | ||
|
|
||
| sudo chmod -R a+rw $BUILDINFO_PATH | ||
| collect_version_files $LOG_VERSION_PATH | ||
| ([ -d $BUILD_VERSION_PATH ] && [ ! -z "$(ls $BUILD_VERSION_PATH/)" ]) && cp -rf $BUILD_VERSION_PATH/* $LOG_VERSION_PATH/ | ||
| mkdir -p $VERSION_SLAVE_PATH | ||
|
|
||
| scripts/versions_manager.py merge -t $VERSION_SLAVE_PATH -b $LOG_VERSION_PATH -e $POST_VERSION_PATH | ||
|
|
||
| rm -rf $BUILD_VERSION_PATH/* | ||
|
|
||
| exit $RET |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| #!/bin/bash | ||
|
|
||
| DOCKER_IMAGE=$1 | ||
| TARGET_PATH=$2 | ||
|
|
||
| [ -z "$TARGET_PATH" ] && TARGET_PATH=./target | ||
|
|
||
| DOCKER_IMAGE_NAME=$(echo $DOCKER_IMAGE | cut -d: -f1) | ||
| DOCKER_CONTAINER=$DOCKER_IMAGE_NAME | ||
| TARGET_VERSIONS_PATH=$TARGET_PATH/versions/dockers/$DOCKER_IMAGE_NAME | ||
|
|
||
| [ -d $TARGET_VERSIONS_PATH ] && rm -rf $TARGET_VERSIONS_PATH | ||
| mkdir -p $TARGET_VERSIONS_PATH | ||
|
|
||
| export DOCKER_CLI_EXPERIMENTAL=enabled | ||
|
|
||
| # Remove the old docker container if existing | ||
| if docker container inspect $DOCKER_IMAGE > /dev/null 2>&1; then | ||
| docker container rm $DOCKER_IMAGE > /dev/null | ||
| fi | ||
| docker create --name $DOCKER_CONTAINER --entrypoint /bin/bash $DOCKER_IMAGE | ||
| docker cp -L $DOCKER_CONTAINER:/etc/os-release $TARGET_VERSIONS_PATH/ | ||
| docker cp -L $DOCKER_CONTAINER:/usr/local/share/buildinfo/pre-versions $TARGET_VERSIONS_PATH/ | ||
| docker cp -L $DOCKER_CONTAINER:/usr/local/share/buildinfo/post-versions $TARGET_VERSIONS_PATH/ | ||
| docker container rm $DOCKER_CONTAINER |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| #!/bin/bash | ||
|
|
||
| TARGET=$1 | ||
| FILESYSTEM_ROOT=$2 | ||
| VERSIONS_PATH=$TARGET/versions/host-image | ||
|
|
||
| mkdir -p $VERSIONS_PATH | ||
|
|
||
| sudo LANG=C chroot $FILESYSTEM_ROOT post_run_buildinfo | ||
|
|
||
| cp -r $FILESYSTEM_ROOT/usr/local/share/buildinfo/pre-versions $VERSIONS_PATH/ | ||
| cp -r $FILESYSTEM_ROOT/usr/local/share/buildinfo/post-versions $VERSIONS_PATH/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| #!/bin/bash | ||
|
|
||
| BUILDINFO_PATH=src/sonic-build-hooks | ||
|
|
||
| BUILDINFO_CONFIG=$BUILDINFO_PATH/buildinfo/config/buildinfo.config | ||
|
|
||
| mkdir -p $BUILDINFO_PATH/buildinfo/config | ||
|
|
||
| echo "PACKAGE_URL_PREFIX=$PACKAGE_URL_PREFIX" > $BUILDINFO_CONFIG | ||
| echo "SONIC_VERSION_CONTROL_COMPONENTS=$SONIC_VERSION_CONTROL_COMPONENTS" >> $BUILDINFO_CONFIG |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.