Skip to content

06.Version Cache - Support for wget #14612

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Praveen-Brcm merged 1 commit into from
Jun 5, 2023
Merged

06.Version Cache - Support for wget #14612

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 90 additions & 20 deletions src/sonic-build-hooks/scripts/buildinfo_base.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,57 +152,127 @@ download_packages()
{
local parameters=("$@")
declare -A filenames
declare -A SRC_FILENAMES
local url=
local real_version=
local SRC_FILENAME=
local DST_FILENAME=

for (( i=0; i<${#parameters[@]}; i++ ))
do
local para=${parameters[$i]}
local nexti=$((i+1))
if [[ "$para" == *://* ]]; then
if [[ "$para" =~ ^-[^-]*o.* || "$para" =~ ^-[^-]*O.* ]]; then
Copy link
Copy Markdown
Contributor

@k-v1 k-v1 May 31, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also possible:

  1. wget www.url.com/file.tar.gz -P directory
  2. wget www.url.com/file.tar.gz -O directory/filename.tar.gz
  3. (only for curl) curl www.url.com/file.tar.gz > filename.tar.gz
  4. wget www.url.com/file.tar.gz -O "file name.tar.gz" - not sure, but probably need to recheck this too.
  5. curl -O www.url.com/file.tar.gz
  6. curl -O www.url.com/file.tar.gz --output-dir test

I'm not sure should we try to fix this or not. I didn't find such a code in current SONiC codebase.
Maybe @xumia can help here and say should we fix or ignore this.

Copy link
Copy Markdown
Contributor

@k-v1 k-v1 May 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more example: curl --output:
https://github.com/sonic-net/sonic-buildimage/blob/6ebad6f8ed2220d2e8eab5aa991c3677bcfe4a1f/platform/mellanox/docker-syncd-mlnx-rpc/Dockerfile.j2#LL68C1-L68C1

Copy link
Copy Markdown
Contributor Author

@Kalimuthu-Velappan Kalimuthu-Velappan Jun 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we ran out of time for the current release, we can take this up on the subsequent release.
Please approve the PR if there are no further changes.

Copy link
Copy Markdown
Contributor

@k-v1 k-v1 Jun 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can ignore this for now and retest again when all parts of vcache are merged.

Copy link
Copy Markdown
Collaborator

@xumia xumia Jun 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree to ignore for now. It may be a little hard to cover all the scenarios, covering the current used scenarios should be good enough.

DST_FILENAME="${parameters[$nexti]}"
elif [[ "$para" == *://* ]]; then
local url=$para
local real_version=

# Skip to use the proxy, if the url has already used the proxy server
if [[ $url == ${URL_PREFIX}* ]]; then
if [[ ! -z "${URL_PREFIX}" && $url == ${URL_PREFIX}* ]]; then
continue
fi
local result=0
WEB_CACHE_PATH=${PKG_CACHE_PATH}/web
mkdir -p ${WEB_CACHE_PATH}
local WEB_FILENAME=$(echo $url | awk -F"/" '{print $NF}' | cut -d? -f1 | cut -d# -f1)
if [ -z "${DST_FILENAME}" ];then
DST_FILENAME="${WEB_FILENAME}"
fi
local VERSION=$(grep "^${url}=" $WEB_VERSION_FILE | awk -F"==" '{print $NF}')
if [ ! -z "${VERSION}" ]; then

if [ "$ENABLE_VERSION_CONTROL_WEB" == y ]; then
if [ ! -z "$(get_version_cache_option)" ]; then
SRC_FILENAME=${WEB_CACHE_PATH}/${WEB_FILENAME}-${VERSION}.tgz
if [ -f "${SRC_FILENAME}" ]; then
log_info "Loading from web cache URL:${url}, SRC:${SRC_FILENAME}, DST:${DST_FILENAME}"
cp "${SRC_FILENAME}" "${DST_FILENAME}"
touch "${SRC_FILENAME}"
continue
fi
fi
fi
fi

if [ "$ENABLE_VERSION_CONTROL_WEB" == y ]; then
local version=
local filename=$(echo $url | awk -F"/" '{print $NF}' | cut -d? -f1 | cut -d# -f1)
[ -f $WEB_VERSION_FILE ] && version=$(grep "^${url}=" $WEB_VERSION_FILE | awk -F"==" '{print $NF}')
if [ -z "$version" ]; then
log_err "Warning: Failed to verify the package: $url, the version is not specified"
continue
fi

local version_filename="${filename}-${version}"
local proxy_url="${PACKAGE_URL_PREFIX}/${version_filename}"
local url_exist=$(check_if_url_exist $proxy_url)
if [ "$url_exist" == y ]; then
parameters[$i]=$proxy_url
filenames[$version_filename]=$filename
real_version=$version
log_err "Warning: Failed to verify the package: $url, the version is not specified" 1>&2
real_version=$(get_url_version $url)
else
real_version=$(get_url_version $url) || { echo "get_url_version $url failed"; exit 1; }
if [ "$real_version" != "$version" ]; then
log_err "Warning: Failed to verify url: $url, real hash value: $real_version, expected value: $version_filename"
continue

local version_filename="${filename}-${version}"
local proxy_url="${PACKAGE_URL_PREFIX}/${version_filename}"
local url_exist=$(check_if_url_exist $proxy_url)
if [ "$url_exist" == y ]; then
parameters[$i]=$proxy_url
filenames[$version_filename]=$filename
real_version=$version
else
real_version=$(get_url_version $url) || { echo "get_url_version $url failed"; exit 1; }
if [ "$real_version" != "$version" ]; then
log_err "Failed to verify url: $url, real hash value: $real_version, expected value: $version_filename" 1>&2
fi
fi
fi
else
real_version=$(get_url_version $url) || { echo "get_url_version $url failed"; exit 1; }
fi

# ignore md5sum for string ""
# echo -n "" | md5sum == d41d8cd98f00b204e9800998ecf8427e
[[ $real_version == "d41d8cd98f00b204e9800998ecf8427e" ]] || echo "$url==$real_version" >> ${BUILD_WEB_VERSION_FILE}
[[ $real_version == "d41d8cd98f00b204e9800998ecf8427e" ]] || {

VERSION=${real_version}
local SRC_FILENAME="${WEB_CACHE_PATH}/${WEB_FILENAME}-${VERSION}.tgz"
SRC_FILENAMES[${DST_FILENAME}]="${SRC_FILENAME}"

echo "$url==$real_version" >> ${BUILD_WEB_VERSION_FILE}
sort ${BUILD_WEB_VERSION_FILE} -o ${BUILD_WEB_VERSION_FILE} -u &> /dev/null
}
fi
done

# Skip the real command if all the files are loaded from cache
local result=0
if [[ ! -z "$(get_version_cache_option)" && ${#SRC_FILENAMES[@]} -eq 0 ]]; then
return $result
fi

$REAL_COMMAND "${parameters[@]}"
local result=$?
result=$?

#Return if there is any error
if [ ${result} -ne 0 ]; then
exit ${result}
fi

for filename in "${!filenames[@]}"
do
[ -f "$filename" ] && mv "$filename" "${filenames[$filename]}"
if [ -f "$filename" ] ; then
mv "$filename" "${filenames[$filename]}"
fi
done

if [[ -z "$(get_version_cache_option)" ]]; then
return $result
fi

#Save them into cache
for DST_FILENAME in "${!SRC_FILENAMES[@]}"
do
SRC_FILENAME="${SRC_FILENAMES[${DST_FILENAME}]}"
if [[ ! -e "${DST_FILENAME}" || -e "${SRC_FILENAME}" ]] ; then
continue
fi
FLOCK "${SRC_FILENAME}"
cp "${DST_FILENAME}" "${SRC_FILENAME}"
chmod -f 777 "${SRC_FILENAME}"
FUNLOCK "${SRC_FILENAME}"
log_info "Saving into web cache URL:${url}, DST:${SRC_FILENAME}, SRC:${DST_FILENAME}"
done

return $result
Expand Down