[build-hooks] Fix reproducible build: pin apt versions and track autoremove by rustiqly · Pull Request #25552 · sonic-net/sonic-buildimage

rustiqly · 2026-02-18T04:34:08Z

Problem

When ENABLE_VERSION_CONTROL_DEB=y (SONIC_VERSION_CONTROL_COMPONENTS includes deb), the apt-get hook has two issues:

1. Versions from versions-deb are never applied to apt-get arguments (#7502)

check_apt_version() only warns about missing package versions — it never modifies the apt-get install command to pin versions. While the APT preferences mechanism (01-versions-deb → /etc/apt/preferences.d/) provides pinning via Pin-Priority: 999, this doesn't guarantee exact version matching (APT can still pick a different version if the pinned one isn't available).

2. `apt-get autoremove` doesn't capture package versions

The hook captures package versions before purge and remove, but not autoremove. Build dependencies installed via apt-get install and later cleaned up via apt-get autoremove are lost from the version tracking, making builds non-reproducible.

Fix

`pin_apt_versions()` — new function in `buildinfo_base.sh`

When ENABLE_VERSION_CONTROL_DEB=y, rewrites apt-get install foo bar to apt-get install foo=1.2.3 bar=4.5.6 by looking up versions in the versions-deb file.

Only activates when deb version control is enabled (off by default)
Packages already pinned (foo=1.2.3) pass through unchanged
Packages not in versions-deb pass through with existing warning
Works alongside the existing APT preferences mechanism as defense-in-depth

autoremove tracking

Added autoremove to the list of commands that trigger version capture (dpkg-query -W → purge-versions-deb), alongside purge and remove.

Impact

No change for default builds — deb is not in the default SONIC_VERSION_CONTROL_COMPONENTS
When version control IS enabled, packages are now actually pinned to the versions in versions-deb
Intermediate build dependencies removed via autoremove are now tracked

Fixes #7502

mssonicbld · 2026-02-18T04:34:16Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-18T04:34:27Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR updates the sonic-build-hooks apt-get hook path to improve reproducible builds when ENABLE_VERSION_CONTROL_DEB=y, by pinning apt-get install package arguments to the versions recorded in versions-deb and by capturing versions prior to apt-get autoremove.

Changes:

Add pin_apt_versions() helper to rewrite apt-get install args into pkg=version using versions-deb.
Update the apt-get hook to apply pinned args before invoking the real apt-get.
Extend version capture to include apt-get autoremove in addition to purge/remove.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
src/sonic-build-hooks/scripts/buildinfo_base.sh	Adds `pin_apt_versions()` to pin `apt-get install` package arguments based on `versions-deb`.
src/sonic-build-hooks/hooks/apt-get	Applies pinned args during installs and captures versions on `autoremove`.

src/sonic-build-hooks/scripts/buildinfo_base.sh

src/sonic-build-hooks/hooks/apt-get

yxieca

autoremove tracking: Clean, LGTM.

pin_apt_versions(): A few concerns:

Grep injection with + in package names — grep "^${para}==" uses regex, but Debian package names commonly contain + (e.g., g++-10, libstdc++6, libc++1). The + is a regex quantifier, so grep "^g++-10==" matches g-10==, gg-10==, etc. instead of the literal package name. This could silently pin the wrong version or fail to match. Fix: use grep -F with an exact match pattern, e.g., grep -F "${para}==" "$VERSION_FILE" | head -1 | awk -F'==' '{print $2}' — since grep -F treats the pattern as a literal string, + is matched correctly.
set -- $versioned_args loses quoting — The subshell output is re-split on whitespace via unquoted $versioned_args. Package names don't have spaces, but option values could. Minor risk but worth noting.
Options after install keyword — Args like -t bullseye will have -t skipped but bullseye treated as a package name and looked up in versions-deb. It won't match so it passes through safely, but it's a design gap worth a comment in the code.

Item #1 is the actionable one — + in package names is common enough to cause real issues.

mssonicbld · 2026-02-20T01:42:32Z

/azp run Azure.sonic-buildimage

rustiqly · 2026-02-20T01:42:34Z

Thanks for the thorough review @yxieca! All addressed in the latest push:

Grep injection with + in package names — Replaced grep "^${para}==" with awk -F'==' -v pkg="$para" '$1==pkg {print $2; exit}'. This does exact string matching on the package name field, so g++, libstdc++6, etc. are handled correctly.
set -- $versioned_args loses quoting — Now uses read -r -a to split into a proper bash array, then set -- "${versioned_args[@]}" to preserve argument boundaries.
Options after install keyword — Added a comment explaining the behavior (option values like bullseye pass through safely since they won't match in versions-deb).

Also fixed Copilot's catch: VERSION_FILE is now local to avoid mutating the global.

azure-pipelines · 2026-02-20T01:42:43Z

Azure Pipelines successfully started running 1 pipeline(s).

yxieca

All feedback addressed — awk for exact match, proper array quoting, options documented. LGTM.

@xumia @liushilongbuaa — could you review the build-hooks changes? The pin_apt_versions() function modifies the apt-get install args to enforce version pinning from versions-deb when ENABLE_VERSION_CONTROL_DEB=y.

liushilongbuaa · 2026-02-23T03:40:20Z

@yijingyan2 , please review and test.

mssonicbld · 2026-02-23T17:37:15Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-23T17:37:26Z

Azure Pipelines successfully started running 1 pipeline(s).

rustiqly · 2026-02-23T17:37:30Z

Addressed all Copilot review comments in latest incremental commit:

VERSION_FILE not local — Now local in both pin_apt_versions() and check_apt_version()
grep regex injection — Replaced with awk -F'==' -v pkg=... exact string match in both functions
check_apt_version uses = vs == — Fixed to use awk with == delimiter matching versions-deb format
set -- $versioned_args injection — Replaced with read -r -a array + quoted expansion

yijingyan2 · 2026-02-24T03:40:18Z

@yijingyan2 , please review and test.

Hi @rustiqly, I ran a test build with the build option SONIC_VERSION_CONTROL_COMPONENTS=py2,py3,web,git,docker,deb. With your changes, the packages versions can be correctly pinned to those specified in the 'versions-deb' file. But this introduced a new issue.
The versions in versions-deb files for docker/host images may contain +fips e.g. openssh-server==1:10.0p1-7+fips (this is because the images were built with build option ENABLE_FIPS=y). So when building these docker/host images, the version for openssh-server is pinned to 1:10.0p1-7+fips, this causes apt-get install fail with the error message:

E: Version '1:10.0p1-7+fips' for 'openssh-server' was not found

currently, we are using snapshot to pin the package versions. It may cause conflict if we pin the versions based on both versions-deb files and snapshot?
@liushilongbuaa, please add or correct anything as needed. Thanks

mssonicbld · 2026-02-24T03:47:44Z

/azp run Azure.sonic-buildimage

rustiqly · 2026-02-24T03:47:45Z

Good catch @yijingyan2! You're right — when versions-deb is generated from a FIPS build, packages like openssh-server get pinned to versions with a +fips suffix that don't exist in non-FIPS repos.

I just pushed a fix: pin_apt_versions() now strips the +fips suffix from the version string when ENABLE_FIPS is not y. This way:

FIPS builds: version used as-is (e.g. 1:10.0p1-7+fips)
Non-FIPS builds: suffix stripped (e.g. 1:10.0p1-7)

Regarding snapshot vs versions-deb conflict — they serve different purposes: snapshot pins the repo state (which packages are available), while versions-deb pins the installed versions. They should be complementary, not conflicting. But if you see issues there, happy to discuss further.

Could you re-test with this latest commit?

azure-pipelines · 2026-02-24T03:47:55Z

Azure Pipelines successfully started running 1 pipeline(s).

yijingyan2 · 2026-02-24T03:59:59Z

Good catch @yijingyan2! You're right — when versions-deb is generated from a FIPS build, packages like openssh-server get pinned to versions with a +fips suffix that don't exist in non-FIPS repos.

I just pushed a fix: pin_apt_versions() now strips the +fips suffix from the version string when ENABLE_FIPS is not y. This way:

FIPS builds: version used as-is (e.g. 1:10.0p1-7+fips)

Non-FIPS builds: suffix stripped (e.g. 1:10.0p1-7)

Regarding snapshot vs versions-deb conflict — they serve different purposes: snapshot pins the repo state (which packages are available), while versions-deb pins the installed versions. They should be complementary, not conflicting. But if you see issues there, happy to discuss further.

Could you re-test with this latest commit?

Hi @rustiqly, if ENABLE_FIPS=y, the apt-get install openssh-server=1:10.0p1-7+fips would also fail, because there are no official versions of +fips from Debian, the +fips version is locally rebuilt.
there is another issue regarding different arch. When building image like mellanox (amd64), it failed with error message:

E: Version '3.11-4+b1' for 'grep' was not found
E: Version '4.9-2+b1' for 'sed' was not found

The versions for grep and sed are pinned to versions that are only available in arm64

rustiqly · 2026-02-24T04:47:17Z

Great catches @yijingyan2, both issues are real:

+fips suffix: You're right — even with ENABLE_FIPS=y, the +fips packages are locally rebuilt and won't exist in Debian repos. Fixed: now stripping +fips unconditionally.
Cross-arch versions: The +b1/+b2 binary NMU revisions are arch-specific, so versions pinned on arm64 can fail on amd64. Fixed: pin_apt_versions() now does an apt-cache show check before pinning — if the version isn't available for the current arch, it falls back to unpinned.

Pushed both fixes. Could you re-test?

mssonicbld · 2026-02-24T04:47:18Z

/azp run Azure.sonic-buildimage

mssonicbld · 2026-03-05T15:01:58Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-05T15:02:11Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-06T15:02:15Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-06T15:02:29Z

Azure Pipelines successfully started running 1 pipeline(s).

rustiqly · 2026-03-08T23:20:37Z

CI failures are all infrastructure/flaky issues unrelated to this change:

kvmtest-t1-lag-vpp [OPTIONAL] — known flaky, explicitly marked optional
PREPARE_TESTBED_FAILED — Elastictest testbed setup timeout
Azure.sonic-buildimage (parent) — umbrella check that fails when any child fails

Could a maintainer please re-trigger CI? Thank you!

mssonicbld · 2026-03-09T14:01:44Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-09T14:01:57Z

Azure Pipelines successfully started running 1 pipeline(s).

yejianquan

LGTM. Three solid fixes:

grep pattern fixed from ^pkg= to ^pkg== matching the actual versions-deb format
autoremove added to version capture -- intermediate build deps were previously lost
+fips suffix stripping for locally-rebuilt FIPS packages

Also good: VERSION_FILE made local to avoid leaking into global scope.

🤖 Posted by DevAce, Jianquan's AI Agent, on his behalf.

StormLiangMS · 2026-03-09T14:45:58Z

@liushilongbuaa ms_conflict.

mssonicbld · 2026-03-10T14:03:53Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-10T14:04:06Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-11T14:01:57Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-11T14:02:13Z

Azure Pipelines successfully started running 1 pipeline(s).

StormLiangMS · 2026-03-12T13:50:47Z

/azpw ms_conflict

mssonicbld · 2026-03-13T02:58:11Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-13T02:58:23Z

Azure Pipelines successfully started running 1 pipeline(s).

…remove When ENABLE_VERSION_CONTROL_DEB=y (SONIC_VERSION_CONTROL_COMPONENTS includes 'deb'), the apt-get hook now actively pins package versions from versions-deb instead of only warning about missing versions. Three fixes: 1. Add pin_apt_versions() - rewrites 'apt-get install foo' to 'apt-get install foo=1.2.3' using versions from versions-deb file. Only activates when deb version control is enabled. Packages not in the versions file pass through unchanged (with existing warning). 2. Track autoremove - collect package versions before apt-get autoremove (like we already do for purge/remove) so intermediate build dependencies are captured in purge-versions-deb. 3. Keep existing APT preferences mechanism (Pin-Priority: 999) as defense-in-depth alongside the explicit version pinning. Fixes sonic-net#7502 Signed-off-by: Rustiqly <[email protected]>

Address Copilot review feedback: - Make VERSION_FILE local in check_apt_version() to avoid global mutation - Replace grep '^pkg=' with awk -F'==' exact match (versions-deb uses ==) Signed-off-by: Rustiqly <[email protected]>

When versions-deb is generated from a FIPS build, package versions may contain a +fips suffix (e.g. openssh-server==1:10.0p1-7+fips). When building without ENABLE_FIPS=y, these versions don't exist in the apt repositories, causing 'apt-get install' to fail. Strip the +fips suffix from pinned versions when ENABLE_FIPS is not enabled. Signed-off-by: Rustiqly <[email protected]>

Address review feedback from liushilongbuaa: remove the pin_apt_versions() function and its call in the apt-get hook. Version pinning is already handled by /etc/apt/preferences.d/01-versions-deb (Pin-Priority: 999), generated by update_preference_deb() and installed by pre_run_buildinfo. Having two mechanisms was confusing and hard to debug. Also add +fips suffix stripping in update_preference_deb() so preferences.d handles FIPS packages correctly (previously only pin_apt_versions did this). Signed-off-by: Rustiqly <[email protected]>

mssonicbld · 2026-03-13T14:01:50Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-13T14:02:04Z

Azure Pipelines successfully started running 1 pipeline(s).

lihuay requested review from Copilot and yxieca February 18, 2026 04:38

Copilot started reviewing on behalf of lihuay February 18, 2026 04:39 View session

lihuay requested review from qiluo-msft and xumia February 18, 2026 04:40

Copilot AI reviewed Feb 18, 2026

View reviewed changes

yxieca reviewed Feb 20, 2026

View reviewed changes

rustiqly force-pushed the fix/reproducible-build-apt-version branch from 8604bc6 to d57b140 Compare February 20, 2026 01:42

yxieca requested a review from liushilongbuaa February 20, 2026 01:46

yxieca previously approved these changes Feb 20, 2026

View reviewed changes

rustiqly mentioned this pull request Feb 20, 2026

[Bug] test_retry_count.py failure in KVM CI due to PORT_PHY_ATTR ERR logs sonic-net/sonic-mgmt#22510

Closed

rustiqly force-pushed the fix/reproducible-build-apt-version branch from d57b140 to 487acfa Compare February 23, 2026 16:59

rustiqly dismissed yxieca’s stale review via 68608c8 February 23, 2026 17:37

rustiqly force-pushed the fix/reproducible-build-apt-version branch from 0c1889b to b5ec9ce Compare February 24, 2026 04:47

rustiqly force-pushed the fix/reproducible-build-apt-version branch from b8108d7 to 15f66f2 Compare March 5, 2026 15:01

rustiqly force-pushed the fix/reproducible-build-apt-version branch from 15f66f2 to 25ca39e Compare March 6, 2026 15:00

rustiqly force-pushed the fix/reproducible-build-apt-version branch from 25ca39e to c615ecd Compare March 9, 2026 14:01

yejianquan approved these changes Mar 9, 2026

View reviewed changes

rustiqly force-pushed the fix/reproducible-build-apt-version branch from c615ecd to d336c37 Compare March 10, 2026 14:02

rustiqly force-pushed the fix/reproducible-build-apt-version branch from d336c37 to 3f0f93e Compare March 11, 2026 14:01

rustiqly force-pushed the fix/reproducible-build-apt-version branch from 3f0f93e to 5f48384 Compare March 13, 2026 02:57

rustiqly added 4 commits March 13, 2026 07:01

fix check_apt_version: local VERSION_FILE + awk exact match

c54ecbd

Address Copilot review feedback: - Make VERSION_FILE local in check_apt_version() to avoid global mutation - Replace grep '^pkg=' with awk -F'==' exact match (versions-deb uses ==) Signed-off-by: Rustiqly <[email protected]>

rustiqly force-pushed the fix/reproducible-build-apt-version branch from 5f48384 to d5f2b87 Compare March 13, 2026 14:01

lihuay merged commit c40f76d into sonic-net:master Mar 13, 2026
20 checks passed

Conversation

rustiqly commented Feb 18, 2026

Problem

1. Versions from versions-deb are never applied to apt-get arguments (#7502)

2. apt-get autoremove doesn't capture package versions

Fix

pin_apt_versions() — new function in buildinfo_base.sh

autoremove tracking

Impact

Uh oh!

mssonicbld commented Feb 18, 2026

Uh oh!

azure-pipelines bot commented Feb 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yxieca left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Feb 20, 2026

Uh oh!

rustiqly commented Feb 20, 2026

Uh oh!

azure-pipelines bot commented Feb 20, 2026

Uh oh!

yxieca left a comment

Choose a reason for hiding this comment

Uh oh!

liushilongbuaa commented Feb 23, 2026

Uh oh!

mssonicbld commented Feb 23, 2026

Uh oh!

azure-pipelines bot commented Feb 23, 2026

Uh oh!

rustiqly commented Feb 23, 2026

Uh oh!

yijingyan2 commented Feb 24, 2026

Uh oh!

mssonicbld commented Feb 24, 2026

Uh oh!

rustiqly commented Feb 24, 2026

Uh oh!

azure-pipelines bot commented Feb 24, 2026

Uh oh!

yijingyan2 commented Feb 24, 2026

Uh oh!

rustiqly commented Feb 24, 2026

Uh oh!

mssonicbld commented Feb 24, 2026

Uh oh!

mssonicbld commented Mar 5, 2026

Uh oh!

azure-pipelines bot commented Mar 5, 2026

Uh oh!

mssonicbld commented Mar 6, 2026

Uh oh!

azure-pipelines bot commented Mar 6, 2026

Uh oh!

rustiqly commented Mar 8, 2026

Uh oh!

mssonicbld commented Mar 9, 2026

Uh oh!

azure-pipelines bot commented Mar 9, 2026

Uh oh!

yejianquan left a comment

Choose a reason for hiding this comment

Uh oh!

StormLiangMS commented Mar 9, 2026

Uh oh!

mssonicbld commented Mar 10, 2026

Uh oh!

azure-pipelines bot commented Mar 10, 2026

Uh oh!

2. `apt-get autoremove` doesn't capture package versions

`pin_apt_versions()` — new function in `buildinfo_base.sh`