[build] Fix rsync failure in Docker builds when SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD is enabled#24604
Merged
kperumalbfn merged 5 commits intosonic-net:masterfrom Jan 5, 2026
Conversation
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
DavidZagury
requested review from
saiarcot895
and removed request for
lguohan and
qiluo-msft
Contributor
Author
|
/azpw run Azure.sonic-buildimage |
Collaborator
|
/AzurePipelines run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
|
@saiarcot895 Please help review/approve/merge this change to fix the prod trixie build as well |
Contributor
Author
|
/azpw run Azure.sonic-buildimage |
Collaborator
|
/AzurePipelines run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
|
@liushilongbuaa , @saiarcot895 Can you please help review? |
Contributor
|
/azpw ms_conflict |
saiarcot895
reviewed
Dec 2, 2025
Contributor
saiarcot895
left a comment
saiarcot895
left a comment
There was a problem hiding this comment.
I see Mellanox platform has been updated, what about Broadcom, VS, and Marvell platforms?
Contributor
Author
@saiarcot895 as we cannot test it for their platform we cannot do that for other vendors. |
Collaborator
|
@saiarcot895 , could you please approve if no more comments? |
Collaborator
@saiarcot895 , can you please ? |
Contributor
|
@saiarcot895 Can you please review? Thanks |
Contributor
|
VS can absolutely be tested; it's just a VM running on a system with at least 6GB of free RAM and 2 cores that can be spared. Additionally, because of the specificity of the command, I'd rather not see this diverge between different dockerfiles. This will just mean more maintenance work later when changes need to be made for whatever reason. |
Contributor
|
Actually, one improvement is to make this into a macro in |
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…RD_FOR_BUILD is enabled When building with SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y, Docker multi-stage builds fail during rsync operations with the following error: rsync: [generator] failed to set times on "/changes-to-image": Read-only file system (30) rsync error: some files/attrs were not transferred (code 23) Root Cause: When SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD is enabled, Docker BuildKit treats bind mounts as read-only by default. The rsync command with -a flag (which includes -t for preserving times) attempts to set timestamps on the mounted directory itself, which fails when the mount is read-only. Solution: Add --omit-dir-times flag to rsync commands in Dockerfile.j2 templates to skip setting directory timestamps on the bind-mounted source directory. This prevents rsync from attempting to modify the read-only mount while still preserving all file timestamps and content. Impact: - Fixes builds with SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y - No functional impact on default builds - Updated 37 Dockerfile.j2 templates across dockers/ and platform/ Signed-off-by: david.zagury <[email protected]>
Signed-off-by: david.zagury <[email protected]>
| {%- endmacro %} | ||
|
|
||
| {% macro rsync_from_builder_stage() -%} | ||
| RUN --mount=type=bind,from=base,target=/changes-to-image rsync -axAX --omit-dir-times --no-D --exclude=/sys --exclude=/proc --exclude=/dev --exclude=resolv.conf /changes-to-image/ / |
There was a problem hiding this comment.
Have you considered doing it this way:
FROM scratch
COPY --from=base / /
Would that work too?
Contributor
There was a problem hiding this comment.
@benoit-nexthop That was my initial approach, but because of differences in kernel settings around overlayfs, this could register as all files being changed even if there are no changes. More details in 0b85785.
9 tasks
Collaborator
|
Cherry-pick PR to 202511: #25093 |
jasonbridges
pushed a commit
to jasonbridges/sonic-buildimage
that referenced
this pull request
Jan 22, 2026
…VE_DOCKERD_FOR_BUILD is enabled (sonic-net#24604) [build] Fix rsync failure in Docker builds when SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD is enabled
mssonicbld
added
Included in 202511 Branch
and removed
Created PR to 202511 Branch
labels
8 tasks
DavidZagury
added a commit
to DavidZagury/sonic-buildimage
that referenced
this pull request
Feb 3, 2026
…_NATIVE_DOCKERD_FOR_BUILD is enabled This is the same fix as in sonic-net#24604 to the new otel docker Signed-off-by: david.zagury <[email protected]>
DavidZagury
added a commit
to DavidZagury/sonic-buildimage
that referenced
this pull request
Feb 4, 2026
…_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD is enabled This is the same fix as in sonic-net#24604 to the new restapi-sidecar docker Signed-off-by: david.zagury <[email protected]>
croos12
pushed a commit
to croos12/sonic-buildimage
that referenced
this pull request
Feb 6, 2026
…_NATIVE_DOCKERD_FOR_BUILD is enabled This is the same fix as in sonic-net#24604 to the new otel docker
croos12
pushed a commit
to croos12/sonic-buildimage
that referenced
this pull request
Feb 6, 2026
…_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD is enabled This is the same fix as in sonic-net#24604 to the new restapi-sidecar docker Signed-off-by: david.zagury <[email protected]>
liat-grozovik
pushed a commit
that referenced
this pull request
Feb 18, 2026
…_NATIVE_DOCKERD_FOR_BUILD is enabled (#25328) This is the same fix as in #24604 to the new otel docker - Why I did it When building SONiC with SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y, Docker multi-stage builds fail during rsync operations with the following error: rsync: [generator] failed to set times on "/changes-to-image": Read-only file system (30) rsync error: some files/attrs were not transferred (see previous errors) (code 23) This issue occurs specifically when: Using Docker Engine 20.10.x with BuildKit Running builds inside containers (Docker-in-Docker scenario) Accessing the host Docker daemon via socket mount Using --mount=type=bind in multi-stage Dockerfiles The root cause is that Docker BuildKit creates read-only bind mounts, and rsync with the -a flag (which includes -t for preserving times) attempts to set timestamps on the mounted directory itself, failing on the read-only mount point. This blocks builds in environments using native dockerd for build acceleration. - How I did it Use the defined in dockerfile-macros.j2 on the otel docker - How to verify it Build SONiC using SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y Verify Docker images build successfully Signed-off-by: david.zagury <[email protected]>
FengPan-Frank
pushed a commit
to FengPan-Frank/sonic-buildimage
that referenced
this pull request
Mar 6, 2026
…VE_DOCKERD_FOR_BUILD is enabled (sonic-net#24604) [build] Fix rsync failure in Docker builds when SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD is enabled Signed-off-by: Feng Pan <[email protected]>
FengPan-Frank
pushed a commit
to FengPan-Frank/sonic-buildimage
that referenced
this pull request
Mar 6, 2026
…_NATIVE_DOCKERD_FOR_BUILD is enabled (sonic-net#25328) This is the same fix as in sonic-net#24604 to the new otel docker - Why I did it When building SONiC with SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y, Docker multi-stage builds fail during rsync operations with the following error: rsync: [generator] failed to set times on "/changes-to-image": Read-only file system (30) rsync error: some files/attrs were not transferred (see previous errors) (code 23) This issue occurs specifically when: Using Docker Engine 20.10.x with BuildKit Running builds inside containers (Docker-in-Docker scenario) Accessing the host Docker daemon via socket mount Using --mount=type=bind in multi-stage Dockerfiles The root cause is that Docker BuildKit creates read-only bind mounts, and rsync with the -a flag (which includes -t for preserving times) attempts to set timestamps on the mounted directory itself, failing on the read-only mount point. This blocks builds in environments using native dockerd for build acceleration. - How I did it Use the defined in dockerfile-macros.j2 on the otel docker - How to verify it Build SONiC using SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y Verify Docker images build successfully Signed-off-by: david.zagury <[email protected]> Signed-off-by: Feng Pan <[email protected]>
mssonicbld
added a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Mar 9, 2026
…_NATIVE_DOCKERD_FOR_BUILD is enabled This is the same fix as in sonic-net#24604 to the new otel docker <!-- Please make sure you've read and understood our contributing guidelines: https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md ** Make sure all your commits include a signature generated with `git commit -s` ** If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx" or "resolves #xxxx" Please provide the following information: --> #### Why I did it When building SONiC with `SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y`, Docker multi-stage builds fail during rsync operations with the following error: ``` rsync: [generator] failed to set times on "/changes-to-image": Read-only file system (30) rsync error: some files/attrs were not transferred (see previous errors) (code 23) ``` This issue occurs specifically when: - Using Docker Engine 20.10.x with BuildKit - Running builds inside containers (Docker-in-Docker scenario) - Accessing the host Docker daemon via socket mount - Using `--mount=type=bind` in multi-stage Dockerfiles The root cause is that Docker BuildKit creates read-only bind mounts, and rsync with the `-a` flag (which includes `-t` for preserving times) attempts to set timestamps on the mounted directory itself, failing on the read-only mount point. This blocks builds in environments using native dockerd for build acceleration. ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it Use the defined in dockerfile-macros.j2 on the otel docker #### How to verify it Build SONiC using SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y Verify Docker images build successfully <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 202305 - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [ ] 202511 #### Tested branch (Please provide the tested image version) <!-- - Please provide tested image version - e.g. - [x] 20201231.100 --> - [ ] <!-- image version 1 --> - [ ] <!-- image version 2 --> #### Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> <!-- Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU. --> #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> Signed-off-by: Sonic Build Admin <[email protected]> #### A picture of a cute animal (not mandatory but encouraged)
8 tasks
mssonicbld
added a commit
that referenced
this pull request
Mar 17, 2026
…_NATIVE_DOCKERD_FOR_BUILD is enabled (#25982) This is the same fix as in #24604 to the new otel docker <!-- Please make sure you've read and understood our contributing guidelines: https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md failure_prs.log skip_prs.log Make sure all your commits include a signature generated with `git commit -s` ** If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx" or "resolves #xxxx" Please provide the following information: --> #### Why I did it When building SONiC with `SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y`, Docker multi-stage builds fail during rsync operations with the following error: ``` rsync: [generator] failed to set times on "/changes-to-image": Read-only file system (30) rsync error: some files/attrs were not transferred (see previous errors) (code 23) ``` This issue occurs specifically when: - Using Docker Engine 20.10.x with BuildKit - Running builds inside containers (Docker-in-Docker scenario) - Accessing the host Docker daemon via socket mount - Using `--mount=type=bind` in multi-stage Dockerfiles The root cause is that Docker BuildKit creates read-only bind mounts, and rsync with the `-a` flag (which includes `-t` for preserving times) attempts to set timestamps on the mounted directory itself, failing on the read-only mount point. This blocks builds in environments using native dockerd for build acceleration. ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it Use the defined in dockerfile-macros.j2 on the otel docker #### How to verify it Build SONiC using SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y Verify Docker images build successfully <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 202305 - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [ ] 202511 #### Tested branch (Please provide the tested image version) <!-- - Please provide tested image version - e.g. - [x] 20201231.100 --> - [ ] <!-- image version 1 --> - [ ] <!-- image version 2 --> #### Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> <!-- Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU. --> #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> Signed-off-by: Sonic Build Admin <[email protected]> #### A picture of a cute animal (not mandatory but encouraged)
dprital
pushed a commit
that referenced
this pull request
Mar 19, 2026
…VE_DOCKERD_FOR_BUILD is enabled (#24604) [build] Fix rsync failure in Docker builds when SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD is enabled Signed-off-by: dprital <[email protected]>
dprital
pushed a commit
that referenced
this pull request
Mar 19, 2026
…_NATIVE_DOCKERD_FOR_BUILD is enabled (#25328) This is the same fix as in #24604 to the new otel docker - Why I did it When building SONiC with SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y, Docker multi-stage builds fail during rsync operations with the following error: rsync: [generator] failed to set times on "/changes-to-image": Read-only file system (30) rsync error: some files/attrs were not transferred (see previous errors) (code 23) This issue occurs specifically when: Using Docker Engine 20.10.x with BuildKit Running builds inside containers (Docker-in-Docker scenario) Accessing the host Docker daemon via socket mount Using --mount=type=bind in multi-stage Dockerfiles The root cause is that Docker BuildKit creates read-only bind mounts, and rsync with the -a flag (which includes -t for preserving times) attempts to set timestamps on the mounted directory itself, failing on the read-only mount point. This blocks builds in environments using native dockerd for build acceleration. - How I did it Use the defined in dockerfile-macros.j2 on the otel docker - How to verify it Build SONiC using SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y Verify Docker images build successfully Signed-off-by: david.zagury <[email protected]> Signed-off-by: dprital <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why I did it
When building SONiC with
SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y, Docker multi-stage builds fail during rsync operations with the following error:This issue occurs specifically when:
--mount=type=bindin multi-stage DockerfilesThe root cause is that Docker BuildKit creates read-only bind mounts, and rsync with the
-aflag (which includes-tfor preserving times) attempts to set timestamps on the mounted directory itself, failing on the read-only mount point.This blocks builds in environments using native dockerd for build acceleration.
Work item tracking
How I did it
Added
--omit-dir-timesflag to all rsync commands in Dockerfile.j2 templates that use bind mounts for multi-stage builds.This flag tells rsync to skip setting directory timestamps (not needed for functionality) while still preserving:
The fix is applied to 37 Dockerfile.j2 files:
dockers/directory (common base Docker images used by all platforms)platform/mellanox/directoryHow to verify it
Build SONiC using SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD=y
Verify Docker images build successfully
Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)