Skip to content

Implement ciflow/rocm on Torchtitan #2114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tianyu-l merged 29 commits into from
Dec 13, 2025
Merged

Implement ciflow/rocm on Torchtitan #2114

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
0539dbe
Implement ciflow/rocm.
akashveramd Dec 5, 2025
19a5a15
Removed tracking_issue & ciflow_tracking_issue.
akashveramd Dec 5, 2025
a2a55a7
Removed ciflow/rocm tag.
akashveramd Dec 5, 2025
cc29b39
Added condition to run ROCm workflow when label is added.
akashveramd Dec 9, 2025
4485cd1
Moved dynamic matrix creation to a separate reusable workflow. Added …
akashveramd Dec 10, 2025
132a3eb
Using set instead of write in set-matrix.
akashveramd Dec 10, 2025
75a1d53
Removed trailing slash from 'ciflow/8gpu' label.
akashveramd Dec 10, 2025
8f8d9df
Refactored set-matrix.yaml. Added option to create ROCm matrix for pu…
akashveramd Dec 10, 2025
9417b90
Fixed indentation and Lint in set-matrix.yaml.
akashveramd Dec 10, 2025
924444d
Fixed indentation in set-matrix.yaml.
akashveramd Dec 10, 2025
e2b8f12
Run ROCm workflow only when PR label added.
akashveramd Dec 11, 2025
db8cb50
Changed the name to TRIGGERED_8GPU_LABEL. Setting default value for T…
akashveramd Dec 11, 2025
c07b45d
Minor comment change.
akashveramd Dec 11, 2025
a53e303
re-ordered if conditions in set-matrix.yaml.
akashveramd Dec 11, 2025
b5d35a4
Dummy commit
akashveramd Dec 11, 2025
1b429ca
Separated condition for PR label trigger in set-matrix.yaml.
akashveramd Dec 11, 2025
878e01b
DEBUG:rolled back original if conditions in set-matrix.yaml and added…
akashveramd Dec 11, 2025
b001e4f
Dummy commit
akashveramd Dec 11, 2025
17e7fd5
DEBUG: Using 4 GPU ROCm runner.
akashveramd Dec 11, 2025
e54daee
Dummy commit
akashveramd Dec 12, 2025
255ab3d
Using 8 GPU ROCm runner.
akashveramd Dec 12, 2025
ffd1da3
Corrected indentation in labeler.yml. Removed debug statements from s…
akashveramd Dec 12, 2025
c6822c9
Corrected indentation in pytorch-probot.yml.
akashveramd Dec 12, 2025
4fd8306
Simplify pull_request trigger
huydhn Dec 13, 2025
158571f
Don't run both CUDA and ROCm job when ciflow/8gpu is there
huydhn Dec 13, 2025
a2f63ab
Remove HAS_8GPU_LABEL
huydhn Dec 13, 2025
917886d
Dummy commit
akashveramd Dec 13, 2025
fab557f
DEBUG
akashveramd Dec 13, 2025
968cdbf
Cleanup set-matrix.yaml.
akashveramd Dec 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/pytorch-probot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
tracking_issue: 24422
ciflow_tracking_issue: 64124
ciflow_push_tags:
- ciflow/8gpu
Copy link
Contributor

@tianyu-l tianyu-l Dec 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does ciflow_push_tags do?
why it's called 8gpu?

Copy link
Collaborator Author

@akashveramd akashveramd Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ciflow_push_tags tells the bot which labels should map to git tags that trigger CI workflows.

It's called 8gpu following Huy's suggestion on slack https://pytorch.slack.com/archives/C0A0E8ARXPZ/p1764843858514179

Copy link
Contributor

@tianyu-l tianyu-l Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems adding ciflow/8gpu label in a PR will automatically create a tag ciflow/8gpu/[PR_number]. I say so because I observed https://github.com/pytorch/torchtitan/releases/tag/ciflow%2F8gpu%2F2125

What is this ciflow_push_tags for? Is it for letting the bot know that the above happened and which tag to look at?

Copy link
Collaborator Author

@akashveramd akashveramd Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct. As per my understanding, when we add PR label, it'll automatically create & push tag. Because the ciflow_push_tags config tells the bot to create & push the matching tags so that tag-based CI workflows can run.

Copy link
Contributor

@wwwjn wwwjn Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shat is the bot doing? Is it doing the auto labeling?

Copy link
Collaborator Author

@akashveramd akashveramd Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The .github/pytorch-probot.yml connects labels to tags. When it sees ciflow/8gpu label in PR, it creates the tag. The automatic labeling is handled by .github/labeler.yml. However, currently I don't see the automatic label getting added when I make changes to PR. @wwwjn Do you know if the totrchtitan repo is setup correctly to put this label automatically or do I need any additional changes?

- ciflow/rocm
10 changes: 7 additions & 3 deletions .github/workflows/integration_test_8gpu_features.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ name: 8 GPU Feature Tests
on:
push:
branches: [ main ]
tags:
- ciflow/8gpu/*
Comment on lines +6 to +7
Copy link
Contributor

@tianyu-l tianyu-l Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this for -- why do we need both pull_request and tags?

Copy link
Collaborator Author

@akashveramd akashveramd Dec 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per my understanding the PR workflows and tag workflows are totally independent. Tags provides CI flow, meaning tags can be pushed to trigger CI runs on specific commits even after the PR is closed. They can also be used for versioning releases.
pull_request are used to run workflows when the PR is open.

paths-ignore:
- 'torchtitan/experiments/**'
pull_request:
Expand All @@ -28,20 +30,22 @@ jobs:
# Step 1: Dynamically compute the matrix based on conditions
set-matrix:
runs-on: ubuntu-latest
env:
HAS_8GPU_LABEL: ${{ (github.event_name == 'pull_request') && contains(github.event.pull_request.labels.*.name, 'ciflow/8gpu') }}
outputs:
matrix: ${{ steps.set.outputs.matrix }}
steps:
- id: set
run: |
# Decide which matrix entries to include based on event type
if [[ "${{ github.event_name }}" == "push" && "${{ github.ref }}" == "refs/heads/main" ]] || [[ "${{ github.event_name }}" == "schedule" ]]; then
# Include both CUDA and ROCm
# Runs CUDA and ROCm for push to main, cron schedule and PR label
if [[ "${{ github.event_name }}" == "push" && "${{ github.ref }}" == "refs/heads/main" ]] || [[ "${{ github.event_name }}" == "schedule" ]] || [[ "${HAS_8GPU_LABEL}" == "true" ]]; then
echo '{"include":[
{"name":"cuda","runner":"linux.g5.48xlarge.nvidia.gpu","gpu-arch-type":"cuda","gpu-arch-version":"12.6","docker-image":"torchtitan-ubuntu-20.04-clang12","index-url":"https://download.pytorch.org/whl/nightly/cu126"},
{"name":"rocm","runner":"linux.rocm.gpu.gfx942.8","gpu-arch-type":"rocm","gpu-arch-version":"7.0","docker-image":"torchtitan-rocm-ubuntu-22.04-clang12","index-url":"https://download.pytorch.org/whl/nightly/rocm7.0"}
]}' > matrix.json
# Runs CUDA for normal PR (without PR label)
else
# Include only CUDA
echo '{"include":[
{"name":"cuda","runner":"linux.g5.48xlarge.nvidia.gpu","gpu-arch-type":"cuda","gpu-arch-version":"12.6","docker-image":"torchtitan-ubuntu-20.04-clang12","index-url":"https://download.pytorch.org/whl/nightly/cu126"}
]}' > matrix.json
Expand Down
Loading