feat: improve cse bootstrap latency by deferring non-critical work by awesomenix · Pull Request #8105 · Azure/AgentBaker

awesomenix · 2026-03-16T23:37:36Z

Summary

reduce the CSE bootstrap critical path by starting containerd/kubelet asynchronously and moving non-critical work until after kubelet startup
add explicit containerd and kubelet readiness checks before running work that depends on them
update supporting tests and VHD builder scripts to match the new startup ordering and latency measurement flow

What changed

in cse_main.sh, skip container runtime installation on Azure Linux OS Guard unless an explicit containerd override is provided, pre-warm kubelet, move containerd ulimit configuration earlier, and defer ensureNoDupOnPromiscuBridge plus non-GPU cleanup until after ensureKubelet
in cse_config.sh, switch containerd startup to the non-blocking helper, wait for containerd before artifact streaming / pause image / GPU-driver work, load nf_conntrack, and record TLS bootstrap start time before kubelet so the latency service measures the full kubelet bootstrap window
in cse_helpers.sh and kubelet.service, add checkServiceHealth, waitForContainerdReady, and an ExecStartPre wait on the containerd socket so later work runs only after services are actually ready
in measure-tls-bootstrapping-latency.sh, emit the completed event using the recorded kubelet start timestamp even when kubeconfig already exists or is created during the race window before inotifywait starts listening
in cse_install.sh and packer scripts, move downloaded kubelet/kubectl binaries into place with normalized ownership/permissions, restart containerd where packer validation needs it, and disable containerd during VHD cleanup so images do not carry the service enabled unexpectedly
add an AzureLinux V3 ARM64 e2e scenario, tag the DCGM exporter compatibility scenario as GPU, and update the ShellSpec coverage for the TLS bootstrapping timing changes

Timings

Before

CSE start: +0.000s
kubelet started: +25.000s
node registered: +26.270s
NodeReady: +26.486s

After

CSE start: +0.000s
ensureKubelet done: +6.489s
kubelet started: +7.000s
first kubelet log: +8.151s
node registered: +9.591s
NodeReady: +9.738s
CSE finish: +19.000s

Copilot

Pull request overview

This PR reduces Linux CSE bootstrap critical-path work by deferring non-essential steps until after ensureKubelet, and updates generated pkg/agent/testdata snapshots to reflect the new CSE/custom data output.

Changes:

Defers ensureNoDupOnPromiscuBridge, enableLocalDNS, and non-GPU driver cleanup until after ensureKubelet in cse_main.sh.
Optimizes provisioning/runtime setup by switching kube binary activation to mv + chmod, and reloading only a targeted sysctl file instead of sysctl --system.
Updates VHD cleanup to disable containerd and regenerates pkg/agent/testdata CustomData snapshots.

Reviewed changes

Copilot reviewed 18 out of 75 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`vhdbuilder/packer/cleanup-vhd.sh`	Disables `containerd` during VHD cleanup to avoid shipping images with it enabled.
`parts/linux/cloud-init/artifacts/cse_main.sh`	Defers some non-critical steps until after `ensureKubelet`; skips container runtime install for golden images/OSGuard.
`parts/linux/cloud-init/artifacts/cse_install.sh`	Changes kubelet/kubectl “activation” to `mv` + `chmod` to avoid redundant copy work.
`parts/linux/cloud-init/artifacts/cse_config.sh`	Uses targeted `sysctl -p` and starts kubelet before the TLS bootstrapping latency measurement service.
`pkg/agent/testdata/MarinerV2+Kata/CustomData`	Regenerated snapshot for updated CSE/custom data output.
`pkg/agent/testdata/CustomizedImage/CustomData`	Regenerated snapshot for updated CSE/custom data output.

You can also share your feedback on Copilot code review. Take the survey.

awesomenix · 2026-03-17T04:10:54Z

+    mv "/opt/bin/kubelet-${KUBERNETES_VERSION}" /opt/bin/kubelet
+    mv "/opt/bin/kubectl-${KUBERNETES_VERSION}" /opt/bin/kubectl
+
+    chmod a+x /opt/bin/kubelet /opt/bin/kubectl


this was what was before, keeping it as is

why this change ? I'm not understanding, install was cleaner ? but slower ?

also, why not force the access level ?

also curious about both

install does a copy and not a move.

Operation: It copies the file to the destination. A key difference from cp is that install unlinks (removes) the destination file first if it already exists, which can prevent issues (like an EBUSY error) when replacing a running executable.

i kept the operation as is before chewi made the change to avoid regression, not sure if it was better or worse but just guarenteed to work and no regression.

https://github.com/Azure/AgentBaker/pull/7125/changes#diff-ff0e92780b2c7f35348b62de54b815b2c9919cfd4f6612f43808aace9dc0a134R638

Regression? My change was merged two months ago. There are important reasons to use install over cp, including the one stated above. There are cases where the destination will be an existing symlink, and it is crucial that we replace the symlink, not its target. mv will do that, but I can't remember if there was some other reason why I didn't stick with mv.

@chewi would you be ok to move it back to mv because its definitely seems faster in this case.

Copilot

Pull request overview

This PR aims to reduce Linux CSE bootstrap critical-path latency by deferring non-critical steps until after ensureKubelet, avoiding redundant work (targeted sysctl reload, moving kube binaries), and adjusting VHD build/runtime behaviors around containerd.

Changes:

Reorders CSE provisioning steps so kubelet starts earlier; starts kubelet before measure-tls-bootstrapping-latency.service.
Optimizes provisioning work (targeted sysctl -p, mv+chmod for kube binaries, skip runtime install when golden image already contains it).
Adjusts VHD build scripts/tests to ensure containerd is started when needed and disabled during image cleanup; regenerates pkg/agent/testdata.

Reviewed changes

Copilot reviewed 18 out of 77 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
vhdbuilder/packer/trivy-scan.sh	Sources provision helpers and ensures containerd is started before Trivy operations.
vhdbuilder/packer/test/linux-vhd-content-test.sh	Starts containerd before executing VHD validation tests.
vhdbuilder/packer/cleanup-vhd.sh	Disables containerd during VHD cleanup.
parts/linux/cloud-init/artifacts/cse_main.sh	Defers non-critical steps until after `ensureKubelet`; skips container runtime install on golden images.
parts/linux/cloud-init/artifacts/cse_install.sh	Uses `mv` + `chmod` when activating downloaded kubelet/kubectl.
parts/linux/cloud-init/artifacts/cse_config.sh	Uses targeted `sysctl -p` and starts kubelet before the TLS bootstrapping latency measurement service.
pkg/agent/testdata/MarinerV2+Kata/CustomData	Regenerated snapshot output for MarinerV2+Kata CustomData.
pkg/agent/testdata/CustomizedImage/CustomData	Regenerated snapshot output for CustomizedImage CustomData.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

Copilot reviewed 23 out of 85 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Copilot · 2026-03-23T22:57:20Z

@@ -536,21 +537,37 @@ systemctlEnableAndStartNoBlock() {
        systemctl status $service --no-pager -l > /var/log/azure/$service-status.log || true
        return 1
    fi
+}


systemctlEnableAndStartNoBlock uses systemctl restart --no-block, which only enqueues the job and can return success even if the unit immediately transitions to failed afterward. ensureKubelet relies on this returning non-zero to detect startup failures, so this can mask real failures (kubelet/measure-tls/containerd). Consider adding a short post-start health check (e.g., systemctl is-failed/ActiveState after a delay) or reintroducing an optional delay parameter so callers can fail fast when the unit enters failed.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Devinwong

Thanks for walking through the changes. LGTM

cameronmeissner · 2026-03-24T23:54:27Z

 ExecStartPre=-/sbin/iptables -t nat --numeric --list

 ExecStartPre=/bin/bash /opt/azure/containers/validate-kubelet-credentials.sh
+ExecStartPre=/bin/sh -c 'until [ -S /run/containerd/containerd.sock ]; do sleep 0.1; done'


is this to avoid kubelet going into a bad exponential back-off or something?

cameronmeissner · 2026-03-24T23:56:26Z

 #!/bin/bash -eux

+systemctl daemon-reload
+systemctl disable --now containerd


nit: would prefer we do this in post-install-dependencies.sh - this script is also ran by the image builder service after optimization is complete (it shouldn't anything if this is also ran by the image builder service, though I think it's cleaner to just have this be scoped to our build-specific logic)

cameronmeissner · 2026-03-24T23:58:58Z

        exit $VALIDATION_ERR
    fi

+    checkServiceHealth containerd || exit $ERR_SYSTEMCTL_START_FAIL


nit: it would be better if we had specific exit codes here, that way we can always pinpoint exactly which call to checkServiceHealth is failing

Copilot AI review requested due to automatic review settings March 16, 2026 23:37

awesomenix requested review from AbelHu, Devinwong, YaoC, calvin197, cameronmeissner, djsly, ganeshkumarashok, junjiezhang1997, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, sulixu, surajssd, timmy-wright, yewmsft and zachary-bailey as code owners March 16, 2026 23:37

awesomenix temporarily deployed to test March 16, 2026 23:37 — with GitHub Actions Inactive

Copilot started reviewing on behalf of awesomenix March 16, 2026 23:38 View session

awesomenix changed the title ~~Improve CSE bootstrap latency by deferring non-critical work~~ feat: improve cse bootstrap latency by deferring non-critical work Mar 16, 2026

Copilot AI reviewed Mar 16, 2026

View reviewed changes

awesomenix force-pushed the nishp/tinyimprovements branch from 7e6264c to 192e020 Compare March 17, 2026 04:16

awesomenix temporarily deployed to test March 17, 2026 04:16 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings March 17, 2026 07:30

awesomenix force-pushed the nishp/tinyimprovements branch from 192e020 to c95a094 Compare March 17, 2026 07:30

awesomenix temporarily deployed to test March 17, 2026 07:30 — with GitHub Actions Inactive

Copilot started reviewing on behalf of awesomenix March 17, 2026 07:31 View session

Copilot AI reviewed Mar 17, 2026

View reviewed changes

Comment thread vhdbuilder/packer/test/linux-vhd-content-test.sh Outdated

awesomenix force-pushed the nishp/tinyimprovements branch from 293a28c to 58051a7 Compare March 22, 2026 11:24

awesomenix temporarily deployed to test March 22, 2026 11:24 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings March 22, 2026 12:46

awesomenix force-pushed the nishp/tinyimprovements branch from 58051a7 to f9809eb Compare March 22, 2026 12:46

awesomenix temporarily deployed to test March 22, 2026 12:46 — with GitHub Actions Inactive

Copilot started reviewing on behalf of awesomenix March 22, 2026 12:46 View session

awesomenix force-pushed the nishp/tinyimprovements branch from f9809eb to e3b7f18 Compare March 22, 2026 12:47

awesomenix temporarily deployed to test March 22, 2026 12:47 — with GitHub Actions Inactive

Copilot AI reviewed Mar 22, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/cse_helpers.sh

Comment thread parts/linux/cloud-init/artifacts/cse_config.sh

awesomenix force-pushed the nishp/tinyimprovements branch from e3b7f18 to 7fc7c13 Compare March 22, 2026 12:53

awesomenix temporarily deployed to test March 22, 2026 12:53 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings March 23, 2026 22:52

awesomenix force-pushed the nishp/tinyimprovements branch from 7fc7c13 to fff45dc Compare March 23, 2026 22:52

awesomenix temporarily deployed to test March 23, 2026 22:53 — with GitHub Actions Inactive

Copilot started reviewing on behalf of awesomenix March 23, 2026 22:53 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

awesomenix force-pushed the nishp/tinyimprovements branch from fff45dc to b938c10 Compare March 23, 2026 23:15

awesomenix temporarily deployed to test March 23, 2026 23:15 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings March 23, 2026 23:33

awesomenix force-pushed the nishp/tinyimprovements branch from b938c10 to e9962a7 Compare March 23, 2026 23:33

Copilot AI reviewed Mar 23, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/measure-tls-bootstrapping-latency.sh

Copilot AI reviewed Mar 24, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/measure-tls-bootstrapping-latency.sh

Comment thread parts/linux/cloud-init/artifacts/cse_config.sh

Comment thread parts/linux/cloud-init/artifacts/cse_main.sh

feat: port over bootstrap latency improvements

8e962d4

Devinwong approved these changes Mar 24, 2026

View reviewed changes

cameronmeissner reviewed Mar 24, 2026

View reviewed changes

cameronmeissner approved these changes Mar 24, 2026

View reviewed changes

djsly mentioned this pull request Apr 11, 2026

feat: add CSE timing regression tests for all Linux VHDs (Ubuntu 22.04/24.04, Azure Linux V3) #8284

Merged

djsly mentioned this pull request May 5, 2026

fix: add backward-compatible shims for hotfix overlay on older VHDs #8452

Closed

Conversation

awesomenix commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Timings

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Devinwong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

awesomenix commented Mar 16, 2026 •

edited

Loading