Skip to content

install diff-filter-buildkite-plugin to only build when relevant files changed #400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fatteneder wants to merge 20 commits into
base: main
Choose a base branch
from
Open

install diff-filter-buildkite-plugin to only build when relevant files changed #400

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
af2b633
install diff-filter-build
fatteneder Jun 11, 2025
9ebd2d7
add diff-filter-doc
fatteneder Nov 2, 2024
698c572
update *upload_jobs.yml
fatteneder Jun 11, 2025
70c3fd9
rm diff-filter-doc, but build at least x86_64-linux
fatteneder Dec 2, 2024
8d4b31e
add option to force build using a PR label
fatteneder Dec 2, 2024
3fbc5a2
rename trigger variable
fatteneder Dec 2, 2024
6ff8918
update ignore list
fatteneder Nov 6, 2024
5ef9ed8
add missing BUILD exports
fatteneder Nov 6, 2024
52f2af1
provide bash script as string for docker plugin command
fatteneder Nov 6, 2024
2ab621e
fix an if condition
fatteneder Nov 6, 2024
b8224c2
can we use the step's command instead of docker's?
fatteneder Nov 7, 2024
65dff2f
try without multiline strings
fatteneder Nov 9, 2024
13c10cb
try with cmd.exe if
fatteneder Nov 9, 2024
1322934
set diff-filter's target_branch option conditionally using environmen…
fatteneder Nov 9, 2024
c44f2b7
use grep -E everywhere for compatability reasons
fatteneder Nov 10, 2024
c9a95fb
add an env var for the target branch
fatteneder Nov 11, 2024
dc5ce91
forward env vars to upload script
fatteneder Dec 2, 2024
368cc29
fix varname
fatteneder Dec 2, 2024
e904a52
also trigger build if PR gets merged
fatteneder Dec 2, 2024
4942000
Revert "forward env vars to upload script"
fatteneder Dec 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .buildkite/hooks/post-checkout
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,8 @@ fi
# Export S3_BUCKET and S3_BUCKET_PREFIX, to force our CI to upload to a different S3 target than the actual CI pipeline would.
export S3_BUCKET="julialang-ephemeral"
export S3_BUCKET_PREFIX="julia-buildkite-uploads/bin"

# for PRs from a julia-buildkite repo we have to set the target branch manually
if [[ -n "$(echo ${BUILDKITE_PULL_REQUEST_REPO} | grep -E "julia-buildkite\.git$")" ]]; then
export DIFF_FILTER_TARGET_BRANCH="master"
fi
188 changes: 121 additions & 67 deletions pipelines/main/launch_unsigned_jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,24 @@
# and only need to touch the webui configuration when we need to alter
# something about the privileged steps.

env:
Copy link
Member Author

@fatteneder fatteneder Nov 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only needed as this repo's main branch is called "main" whereas the one of the Julia repo is called "master".

Using a default of DIFF_FILTER_TARGET_BRANCH="" tells the diff-filter-buildkite-plugin to use a PR's merge base as the reference against which changes should be compared (https://github.com/fatteneder/diff-filter-buildkite-plugin/blob/7155804f7ba73a9798b4c4225613ca6a5d7d8c12/hooks/diff-filter-impl#L13-L38).

So if the pipeline run's on forks of Julia's repo, it should do the right thing (although untested) even when PRing against a branch other than master. For PRs against julia-buildkite we always compare against Julia's master by manually setting it in .buildkite/hooks/post-checkout.

DIFF_FILTER_TARGET_BRANCH: ""

common:
- diff-filter-build_plugin: &diff-filter-build
https://github.com/fatteneder/diff-filter-buildkite-plugin#main:
Copy link
Member

@giordano giordano Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this should be moved to this organisation before merge.

name: "TRIGGER_BUILD"
ignore:
- "*.md"
- ".*"
- "julia.spdx.json"
- "CITATION.*"
- "typos.toml"
target_branch: "${DIFF_FILTER_TARGET_BRANCH?}"
- pr-labels_plugin: &pr-labels
sv-oss/github-pr-labels#v0.0.2:
publish-env-var: PULL_REQUEST_LABELS

steps:
- group: "Build"
notify:
Expand All @@ -33,35 +51,45 @@ steps:
# that this script doesn't suddenly break when a new patch release of Julia
# is released.
version: '1.7.3'
- *diff-filter-build
- *pr-labels
commands: |
### Launch Linux build jobs.
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
### Launch Linux build jobs
# Regular:
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_linux.arches \
.buildkite/pipelines/main/platforms/build_linux.yml
# PowerPC (only for Julia prior to 1.12):
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
julia .buildkite/pipelines/main/platforms/launch_powerpc.jl \
.buildkite/pipelines/main/platforms/build_linux.powerpc.arches \
.buildkite/pipelines/main/platforms/build_linux.yml
### Launch macOS build jobs:
### Launch macOS packaging jobs
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_macos.arches \
.buildkite/pipelines/main/platforms/build_macos.yml
### Launch FreeBSD build jobs:
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_freebsd.arches \
.buildkite/pipelines/main/platforms/build_freebsd.yml
### Launch Windows build jobs:
GROUP="Build" \
ALLOW_FAIL="false" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_windows.arches \
.buildkite/pipelines/main/platforms/build_windows.yml
Expand All @@ -79,17 +107,24 @@ steps:
- JuliaCI/external-buildkite#v1:
version: "./.buildkite-external-version"
repo_url: "https://github.com/JuliaCI/julia-buildkite"
- *diff-filter-build
- *pr-labels
commands: |
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/analyzegc.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/doctest.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/pdf_docs/build_pdf_docs.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/embedding.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/trimming.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/llvmpasses.yml
# buildkite-agent pipeline upload .buildkite/pipelines/main/misc/whitespace.yml # Currently runs in GitHub Actions instead of Buildkite
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
if [[ $${BUILD} == 1 ]]; then
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/analyzegc.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/embedding.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/trimming.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/llvmpasses.yml
# buildkite-agent pipeline upload .buildkite/pipelines/main/misc/whitespace.yml # Currently runs in GitHub Actions instead of Buildkite

buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/asan.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/tsan.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/asan.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/sanitizers/tsan.yml
fi
agents:
queue: "julia"
os: "linux"
Expand All @@ -104,38 +139,44 @@ steps:
- JuliaCI/external-buildkite#v1:
version: "./.buildkite-external-version"
repo_url: "https://github.com/JuliaCI/julia-buildkite"
- *diff-filter-build
- *pr-labels
commands: |
export ALLOW_FAIL="false"
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/gcext.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/test_revise.yml
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
if [[ $${BUILD} == 1 ]]; then
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/gcext.yml
buildkite-agent pipeline upload .buildkite/pipelines/main/misc/test_revise.yml

### Launch Linux test jobs.
# Regular:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
# i686-linux-gnu:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.i686.arches \
.buildkite/pipelines/main/platforms/test_linux.i686.yml
### Launch macOS test jobs:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_macos.arches \
.buildkite/pipelines/main/platforms/test_macos.yml
### Launch FreeBSD test jobs:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_freebsd.arches \
.buildkite/pipelines/main/platforms/test_freebsd.yml
### Launch Windows test jobs:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_windows.arches \
.buildkite/pipelines/main/platforms/test_windows.yml
echo "+++ Finished launching test jobs"
### Launch Linux test jobs.
# Regular:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
# i686-linux-gnu:
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.i686.arches \
.buildkite/pipelines/main/platforms/test_linux.i686.yml
### Launch macOS test jobs
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_macos.arches \
.buildkite/pipelines/main/platforms/test_macos.yml
### Launch FreeBSD test jobs
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_freebsd.arches \
.buildkite/pipelines/main/platforms/test_freebsd.yml
### Launch windows test jobs
GROUP="Test" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_windows.arches \
.buildkite/pipelines/main/platforms/test_windows.yml
fi
agents:
queue: "julia"
os: "linux"
Expand All @@ -147,15 +188,22 @@ steps:
- JuliaCI/external-buildkite#v1:
version: "./.buildkite-external-version"
repo_url: "https://github.com/JuliaCI/julia-buildkite"
- *diff-filter-build
- *pr-labels
commands: |
export ALLOW_FAIL="true"
# Launch Linux allowed-to-fail build jobs:
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
# Launch Linux allowed-to-fail build jobs
GROUP="Allow Fail" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_linux.soft_fail.arches \
.buildkite/pipelines/main/platforms/build_linux.yml
# Launch macOS allowed-to-fail build jobs:
# Launch macOS allowed-to-fail build jobs
GROUP="Allow Fail" \
BUILD="$${BUILD}" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/build_macos.soft_fail.arches \
.buildkite/pipelines/main/platforms/build_macos.yml
Expand All @@ -178,35 +226,41 @@ steps:
# that this script doesn't suddenly break when a new patch release of Julia
# is released.
version: '1.7.3'
- *diff-filter-build
- *pr-labels
commands: |
export ALLOW_FAIL="true"

### Launch Linux allowed-to-fail test jobs.
# Regular:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
# PowerPC (only for Julia prior to 1.12):
GROUP="Test" \
julia .buildkite/pipelines/main/platforms/launch_powerpc.jl \
.buildkite/pipelines/main/platforms/test_linux.powerpc.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
### Launch macOS allowed-to-fail test jobs:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_macos.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_macos.yml
### Launch FreeBSD allowed-to-fail jobs:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_freebsd.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_freebsd.yml
### Launch Windows allowed-to-fail test jobs:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_windows.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_windows.yml
FORCE=$(test -n "$(echo $${PULL_REQUEST_LABELS} | grep -E "\bci-force-build\b")" && echo 1 || echo 0)
MERGED=$(test "$${BUILDKITE_PULL_REQUEST}" = "false" && echo 1 || echo 0)
BUILD=$(($${TRIGGER_BUILD} || $${FORCE} || $${MERGED}))
if [[ $${BUILD} == 1 ]]; then
### Launch Linux allowed-to-fail test jobs.
# Regular:
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_linux.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
# PowerPC (only for Julia prior to 1.12):
GROUP="Test" \
julia .buildkite/pipelines/main/platforms/launch_powerpc.jl \
.buildkite/pipelines/main/platforms/test_linux.powerpc.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_linux.yml
### Launch macOS allowed-to-fail test jobs
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_macos.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_macos.yml
### Launch FreeBSD allowed-to-fail jobs
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_freebsd.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_freebsd.yml
### Launch windows allowed-to-fail test jobs
GROUP="Allow Fail" \
bash .buildkite/utilities/arches_pipeline_upload.sh \
.buildkite/pipelines/main/platforms/test_windows.soft_fail.arches \
.buildkite/pipelines/main/platforms/test_windows.yml
fi
agents:
queue: "julia"
os: "linux"
Expand Down
Loading