v0.29.3

v0.29.3 (2025-07-09)

Full Changelog

Warning

Breaking changes in v0.29

Starting v0.29.0 we stopped supporting unclustered setup. If you are using unclustered agent setup, please follow this migration guide.

We no longer automatically append a queue=kubernetes tag. Instead, if queue tags are not explicitly specified when the agent is started, then the controller will pull jobs from the default queue.

Added

• PB-164 part 2: add opt-out flag for imagecheck containers #643 (@zhming0)
• PIPE-1263: set user-agent header #641 (@zhming0)

Internal

• PB-133-follow-up: update cmd description for experimental-job-reservation-support #647 (@zhming0)
• PB-164 part 1: relocating imagecheck related logic to another file #642 (@zhming0)
• PB-133: feature flag for job reservation support #644 (@zhming0)
• chore(deps): Bump github.com/go-playground/validator/v10 from 10.26.0 to 10.27.0 #645 (@dependabot[bot])
• chore(deps): Bump github.com/buildkite/agent/v3 from 3.100.1 to 3.101.0 #646 (@dependabot[bot])

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.29.3
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.29.3
Digest: sha256:fd99d548320e609d2c976f160f159b85e67350ad3d81226ea726d18c91fa742d

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.29.3
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.29.3
Digest: sha256:92c9a9edf74f5a155e4edf27c20b7a721275d2e3154dbb957eb49c38e79c8fc3

Agent

Image: ghcr.io/buildkite/agent:3.101.0
Digest: sha256:ad8e8872a03d2c8b1258c8415e8a2814e239e27cc81757557570b4e486b7a7ec

v0.29.2

v0.29.2 (2025-06-26)

Full Changelog

Warning

Breaking changes in v0.29

Starting v0.29.0 we stopped supporting unclustered setup. If you are using unclustered agent setup, please follow this migration guide.

We no longer automatically append a queue=kubernetes tag. Instead, if queue tags are not explicitly specified when the agent is started, then the controller will pull jobs from the default queue.

Added

• Add simple usage instructions #638 (@sj26)

Fixed

• Precreate build and plugins dirs #637 (@DrJosh9000)

Changed

• chore(deps): Bump github.com/buildkite/agent/v3 from 3.100.0 to 3.100.1 #639 (@dependabot[bot])

Internal

• go.mod: remove mapstructure replace #636 (@DrJosh9000)
• PIPE-1111-part-1: allow running k8s agent stack against local bk/bk #630 (@zhming0)

Dependencies

• chore(deps): Bump the k8s group across 1 directory with 5 updates #640 (@dependabot[bot])
• chore(deps): Bump github.com/go-viper/mapstructure/v2 from 2.2.1 to 2.3.0 #634 (@dependabot[bot])
• chore(deps): Bump gotest.tools/gotestsum from 1.12.2 to 1.12.3 #633 (@dependabot[bot])

Agent changelog

v3.100.1 (2025-06-25)

Full Changelog

Fixed

• Fix regression in pipeline upload with no-interpolation #3359 (@DrJosh9000)

Internal

• Avoid goroutine failing after test #3356 (@DrJosh9000)

Dependencies

• build(deps): bump github.com/buildkite/shellwords from 0.0.0-20180315084142-c3f497d1e000 to 1.0.0 #3352 (@dependabot[bot])
• build(deps): bump github.com/go-chi/chi/v5 from 5.2.1 to 5.2.2 #3353 (@dependabot[bot])
• build(deps): bump the container-images group across 6 directories with 2 updates #3354 (@dependabot[bot])
• build(deps): bump the cloud-providers group with 5 updates #3355 (@dependabot[bot])

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.29.2
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.29.2
Digest: sha256:55b6d53c35f199aba7dc9b07eca8a48f3517577b12cd13b1db715198f5fbde5f

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.29.2
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.29.2
Digest: sha256:dba460b292526b9116d4ff4cb3a579fdae6407d14c8abcb183f53762751dda2b

Agent

Image: ghcr.io/buildkite/agent:3.100.1
Digest: sha256:6152427bdadc8e84e401b97593ba0ff59046e5ef72f0646d34156efbee180003

v0.29.1

v0.29.1 (2025-06-23)

Full Changelog

Warning

Breaking changes in v0.29

Starting v0.29.0 we stopped supporting unclustered setup. If you are using unclustered agent setup, please follow this migration guide.

We no longer automatically append a queue=kubernetes tag. Instead, if queue tags are not explicitly specified when the agent is started, then the controller will pull jobs from the default queue.

Changed

• Link straight to agent tokens #631 (@sj26)
• Upgrade agent to v3.100.0 #632 (@zhming0)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.29.1
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.29.1
Digest: sha256:55bf2096d2e1e299073d55f45f7b0281c32b9c784ae96f66791574288a684217

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.29.1
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.29.1
Digest: sha256:425121143788727c6766f5f84ca88345019525c1d55c303ad820a4e26ef11aae

Agent

Image: ghcr.io/buildkite/agent:3.100.0
Digest: sha256:4b87ac974680ba34fe6784bdb57abd0c7932043bbc66ff42fc47a987518d63ba

v0.29.0

v0.29.0 (2025-06-19)

Full Changelog

Warning

Breaking

In v0.29.0 we stopped supporting unclustered agents. If you are using unclustered agents, please follow this migration guide.

Another breaking change: we no longer automatically append a queue=kubernetes tag. Instead, if queue tags are not explicitly specified when the agent is started, then the controller will pull jobs from the default queue.

• [Breaking] drop support for unclustered setup #611 (@zhming0)
• [BREAKING]: listen to default queue when no queue is specified #615 (@zhming0)

Change

• Output bad yaml when kube api server rejects the request #624 (@nsuma8989)
• Stack error signal reason #627 (@moskyb)
• PB-5: improve exit code propogation for non-command containers #628 (@zhming0)

Fixed

• PIPE-1171: fix all jobs must match agent tags bug + introduce controller ID #626 (@zhming0)
• PIPE-1072-2: fix explicitly setting job default queue causing controller to miss the job #622 (@zhming0)

Internal

• PIPE-1157: Dogfooding buildkite scoped secret for the graphql token #623 (@zhming0)
• PIPE-1115: update development guide #621 (@zhming0)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.29.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.29.0
Digest: sha256:f42bd765e10e59af2ea8cc58689fd61d04c6c65d1fa85e0c8b2ec6262ae410cf

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.29.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.29.0
Digest: sha256:e4b76d971ed538b46ec7c0a2ac50b2fc59e714f24b046204f53305b1800db224

Agent

Image: ghcr.io/buildkite/agent:3.98.2
Digest: sha256:d9377fd83c89828d50a2e82a1a9209b888cc9c0d8ccef12b55632e6a6fd9228e

v0.28.2

v0.28.2 (2025-06-19)

Full Changelog

Fixed

• PIPE-1171: fix all jobs must match agent tags bug + introduce ID config (@zhming0)

Internal

• PIPE-1157: Dogfooding buildkite scoped secret for the graphql token (@zhming0)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.28.2
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.28.2
Digest: sha256:82ca22f9e620185e06589fabeba01438ca5c01cbc559dd8585054e5d1f6c8451

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.28.2
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.28.2
Digest: sha256:e64bb010c00527e4d181407573aea6ac7429e47a0cf93c07c86de1e63e451248

Agent

Image: ghcr.io/buildkite/agent:3.98.1
Digest: sha256:e57c0c8a003895bfe3bb0a10d15ee892349a3c31b4606ebea17fe92af662fae0

v0.28.1

v0.28.1 (2025-06-12)

Full Changelog

Fixed

• fix checkout skip crash + container-0 double handling #620 (@zhming0)
• Fix missing hooks and plugins volume mounts #619 (@DrJosh9000)

Internal

• Add clean-up-job.yaml for Buildkite kubernetes job clean up #605 (@Damilola-obasa)
• Add integration test for default queue handling #616 (@DrJosh9000)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.28.1
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.28.1
Digest: sha256:7999ea1c40919821e95d2e4bce37c5196ca7713ef0f76c0c0386484745c9b992

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.28.1
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.28.1
Digest: sha256:628292a963972a8ad53f9a3a562da0a323c64a21eeab288cd734642cf1835525

Agent

Image: ghcr.io/buildkite/agent:3.98.1
Digest: sha256:e57c0c8a003895bfe3bb0a10d15ee892349a3c31b4606ebea17fe92af662fae0

v0.28.0

v0.28.0 (2025-06-05)

Full Changelog

Warning

A lot has changed since v0.27.0. Please take note of the following Important Changes if you are upgrading.

• The GraphQL token is no longer needed or used. You may remove it from your config. Only an agent token is required.
• The organisation slug and cluster UUID are no longer needed. You may remove org and cluster-uuid from your config. They are now inferred from the agent token.
• Some environment variables no longer have an effect when specified in a pipeline upload or new build options. Previously, certain agent behaviours could be controlled this way, which was inconsistent with how the agent operates in other environments. Examples include BUILDKITE_GIT_CLONE_FLAGS andBUILDKITE_GIT_FETCH_FLAGS. Instead, you can supply equivalent options in agent-config or default-checkout-params in the agent-stack-k8s config, or use the kubernetes plugin in a pipeline to supply equivalent options in checkout-params per step. If you are relying on a particular environment variable that isn't supported by the available config options, please let us know.
• Buildkite Agent v3.98.1 (or later) is required. If you are using a custom agent image, please rebuild with the latest agent.

Since v0.28.0-beta6

Changed

• Mark config.org and config.cluster-uuid as deprecated #614 (@DrJosh9000)
• PIPE-1071: remove cluster uuid from stack parameter #610 (@zhming0)
• PIPE-1084: remove org as stack parameter #609 (@zhming0)
• chore(deps): Bump github.com/buildkite/agent/v3 from 3.98.0 to 3.98.1 #612 (@dependabot[bot])

Fixed

• pod-spec-patch without containers specified removes all containers from job pod #608 (@sj26)
• Fix note alert gfm syntax in readme #607 (@sj26)
• Fix default queue handling #604 (@DrJosh9000)

Internal

• Fix TestCancelCheckerDeletePod flakiness #603 (@zhming0)
• Increase test queue name uniqueness and improve test cleanup robustness #613 (@zhming0)

v0.28.0-beta6

Changed

Pods for jobs that have been cancelled prior to the agent starting are now deleted directly, instead of being evicted.

• PG-80: Replace evicePod with forcefullyDeletePod #600 (@zhming0)

Fixed

The change which broke cleanup of pods with sidecar containers has been reverted.

• Revert PR-591: fixing dangling sidecar container issue #602 (@zhming0)

Checkout-params and default-checkout-params should once again be properly applied.

• PG-148: ensure checkout-params is properly passed to checkout container #601 (@zhming0)

v0.28.0-beta5

Changed

• Use new agent kubernetes-bootstrap #595 (@DrJosh9000)
• Send SIGTERM to any application (non-init) sidecars for cleanup #591 (@petetomasik)
• Offloading documentation into the official documentation #594 (@karensawrey)

Fixed

• Ensure controller agent tags match job tags 100% #596 (@petetomasik)

Dependency updates

• chore(deps): Bump github.com/buildkite/agent/v3 from 3.97.0 to 3.97.2 #598 (@dependabot[bot])
• chore(deps): Bump the k8s group with 3 updates #599 (@dependabot[bot])
• chore(deps): Bump github.com/Khan/genqlient from 0.8.0 to 0.8.1 #597 (@dependabot[bot])
• chore(deps): Bump gotest.tools/gotestsum from 1.12.1 to 1.12.2 #592 (@dependabot[bot])

v0.28.0-beta4

Changed

• Bump sleeps for flakey tests #590 (@CerealBoy)

Fixed

• Fix name of buildkite_deduper_onupdate_events_total metric #588 (@DrJosh9000)
• Fix regression preventing evicting pods for cancelled jobs #589 (@DrJosh9000)

v0.28.0-beta3

Added

• Implement configurable resource limits for imagecheck containers #580 (@petetomasik)
• feat: pave road to custom annotations #582 (@scadu)
• Add Prometheus metrics docs page + generator #581 (@DrJosh9000)

Changed

• Update docs for v0.28 GraphQL removal #579 (@DrJosh9000)

Fixed

• Add prometheus-port to schema (it was missing) #585 (@DrJosh9000)
• Configurable work queue limit (avoids memory exhaustion due to unbounded temporary storage of jobs) #584 (@DrJosh9000)

v0.28.0-beta2

Added

• Setup SSH Keys in container-0 #561 (@lizrabuya)
• Allow extra EnvVar on controller container #571 (@petetomasik)
• Document use of nodeSelector and nodeName PodSpec fields #569 (@petetomasik)

Changed

• Deprecate GraphQL token #572 (@DrJosh9000)
• Respect Retry-After header #577 (@DrJosh9000)

Fixed

• Fix resource limits example #576 (@petetomasik)

Dependency updates

• chore(deps): Bump github.com/buildkite/agent/v3 from 3.96.0 to 3.97.0 #574 (@dependabot[bot])

v0.28.0-beta1

Added

• feat: New APIs used to fetch jobs #562 (@DrJosh9000)

Changed

• Update config defaults + docs #570 (@swaller-bk)
• SUP-3454 - Update PodSpec docs, small fix/cleanup #568 (@petetomasik)
• SUP-3717 - Docs/more updates #567 (@petetomasik)
• Validation for the container image name #565 (@nsuma8989)

Internal

• chore(deps): Bump github.com/buildkite/agent/v3 from 3.95.1 to 3.96.0 #564 (@dependabot[bot])

Buildkite Agent changes

v3.98.1 (2025-06-04)

Full Changelog

Fixed

• Gracefully Handle Missing GitHub PR refs/pull/%s/head in Checkout #3294 (@123sarahj123)
• Fix bootstrap subprocess handling #3331 (@DrJosh9000)
• Reduce git fetch from twice to once for typical Github PR build #3327 (@zhming0)
• Set job log tempfile permissions to 644 (was 600) #3330 (@moskyb)

Internal

• Tag tests with os / arch #3326 (@catkins)

v3.98.0 (2025-05-27)

Full Changelog

Added

• Add build URL to log fields #3317 (@ChrisBr)
• Add kubernetes-bootstrap subcommand #3306, #3314, #3316 (@DrJosh9000)

Fixed

• Fix redactor add --format json help string #3322 (@francoiscampbell)

Dependency updates

• #3320, #3318, #3319, #3323, #3321 (@dependabot[bot])

[v3.97.2](https://github.com/bui...

v0.28.0-beta6

v0.28.0-beta6 (2025-05-28)

Full Changelog

Warning

This is a beta release.

(Same note as beta 5) The most significant change is the rearrangement of environment variables among containers. The bulk of env vars are now communicated across a socket shared between the containers in a pod, rather than through the Kubernetes resources. Note that this new mechanism depends on using the latest Buildkite Agent release (v3.98.0). If you use a custom agent image, you must rebuild it to use Buildkite Agent v3.98.0 or later in order to use this (or later versions) or your builds will fail.

As with betas 2 through k, this release exclusively uses a new REST API and deprecates use of the GraphQL API. As this is a new API, we may make breaking changes to it as we receive feedback from real-world use. As such, if you do use it, we suggest upgrading to the full release once it is available, as APIs the beta relies on may be removed in the future.

Changed

Pods for jobs that have been cancelled prior to the agent starting are now deleted directly, instead of being evicted.

• PG-80: Replace evicePod with forcefullyDeletePod #600 (@zhming0)

Fixed

The change which broke cleanup of pods with sidecar containers has been reverted.

• Revert PR-591: fixing dangling sidecar container issue #602 (@zhming0)

Checkout-params and default-checkout-params should once again be properly applied.

• PG-148: ensure checkout-params is properly passed to checkout container #601 (@zhming0)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.28.0-beta6
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.28.0-beta6
Digest: sha256:a689b70aa913729f818868ff78ae876beb095deed51951aa3a3c5b9c9d33af70

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.28.0-beta6
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.28.0-beta6
Digest: sha256:6746609e8133cf598c1f5006160fa3d93de75d32ed775b4adeb1e21a6bb75991

Agent

Image: ghcr.io/buildkite/agent:3.98.0
Digest: sha256:cdf402ab62d0c4796a209f20d1032cc08f2da325c45f5185b36962a493dbe4a8

v0.28.0-beta5

v0.28.0-beta5 (2025-05-27)

Full Changelog

Warning

This is a beta release.

The most significant change is the rearrangement of environment variables among containers. The bulk of env vars are now communicated across a socket shared between the containers in a pod, rather than through the Kubernetes resources. Note that this new mechanism depends on using the latest Buildkite Agent release (v3.98.0). If you use a custom agent image, you must rebuild it to use Buildkite Agent v3.98.0 or later in order to use this (or later versions) or your builds will fail.

As with betas 2 through 4, this release exclusively uses a new REST API and deprecates use of the GraphQL API. As this is a new API, we may make breaking changes to it as we receive feedback from real-world use. As such, if you do use it, we suggest upgrading to the full release once it is available, as APIs the beta relies on may be removed in the future.

Changed

• Use new agent kubernetes-bootstrap #595 (@DrJosh9000)
• Send SIGTERM to any application (non-init) sidecars for cleanup #591 (@petetomasik)
• Offloading documentation into the official documentation #594 (@karensawrey)

Fixed

• Ensure controller agent tags match job tags 100% #596 (@petetomasik)

Dependency updates

• chore(deps): Bump github.com/buildkite/agent/v3 from 3.97.0 to 3.97.2 #598 (@dependabot[bot])
• chore(deps): Bump the k8s group with 3 updates #599 (@dependabot[bot])
• chore(deps): Bump github.com/Khan/genqlient from 0.8.0 to 0.8.1 #597 (@dependabot[bot])
• chore(deps): Bump gotest.tools/gotestsum from 1.12.1 to 1.12.2 #592 (@dependabot[bot])

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.28.0-beta5
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.28.0-beta5
Digest: sha256:1917525c96a4b13725e7febda52607c9f6f8e2ace469936668be206b67267d70

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.28.0-beta5
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.28.0-beta5
Digest: sha256:855327be87ea6519f68704850329e5268d7f8042949fbad573ebd39def3e8dc0

Agent

Image: ghcr.io/buildkite/agent:3.98.0
Digest: sha256:cdf402ab62d0c4796a209f20d1032cc08f2da325c45f5185b36962a493dbe4a8

v0.28.0-beta4

v0.28.0-beta4 (2025-05-08)

Full Changelog

Warning

This is a beta release. Similar to beta 2, this release exclusively uses a new REST API and deprecates use of the GraphQL API. As this is a new API, we may make breaking changes to it as we receive feedback from real-world use. As such, if you do use it, we suggest upgrading to the full release once it is available, as APIs the beta relies on may be removed in the future.

Changed

• Bump sleeps for flakey tests #590 (@CerealBoy)

Fixed

• Fix name of buildkite_deduper_onupdate_events_total metric #588 (@DrJosh9000)
• Fix regression preventing evicting pods for cancelled jobs #589 (@DrJosh9000)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.28.0-beta4
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.28.0-beta4
Digest: sha256:3efdb5af8f0cec518a61ac94875bd7bcf9a3c988d88ce70ccb12d8eeb112d4ea

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.28.0-beta4
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.28.0-beta4
Digest: sha256:9b09bd59dcc58761674dc45bf71b11e4ae6d5e8393061c46219618b1d8f79d76

Agent

Image: ghcr.io/buildkite/agent:3.97.0
Digest: sha256:bdae36b65b2b7c0a6f2c4eb7ec2e86c5499e73eaca2a30336bc0ea5ae7a43d01