Implement ciflow/rocm #2114

akashveramd · 2025-12-05T03:18:58Z

In this PR, I implemented ciflow/rocm. The changes are part of integration_test_8gpu_features.yaml. The workflow still supports running on pull_request (without any PR label) for CUDA. However, along with push to main and cron schedule, with the ciflow/8gpu label added to PR, the workflow runs for both CUDA & ROCm.

.github/pytorch-probot.yml

tianyu-l · 2025-12-05T09:34:10Z

with the ciflow/8gpu label added to PR, the workflow runs for both CUDA & ROCm.

Can this label be automatically added? If not, what happens if it is not manually added -- is rocm test skipped?

.github/pytorch-probot.yml

akashveramd · 2025-12-05T20:02:56Z

with the ciflow/8gpu label added to PR, the workflow runs for both CUDA & ROCm.

Can this label be automatically added? If not, what happens if it is not manually added -- is rocm test skipped?

If the label is not added manually then the ROCm tests will not run. CUDA tests can still run normally.
@huydhn: Please correct me if I am wrong, but we wanted to add label manually by code owners due to security risk, similar to how it works in PyTorch?

huydhn · 2025-12-06T08:07:29Z

If the label is not added manually then the ROCm tests will not run. CUDA tests can still run normally. @huydhn: Please correct me if I am wrong, but we wanted to add label manually by code owners due to security risk, similar to how it works in PyTorch?

For the context, ciflow bot only works if the PR author has write permission to the repo or if the CI has already been approved to run https://github.com/pytorch/test-infra/blob/main/torchci/lib/bot/ciflowPushTrigger.ts#L176-L183. This policy applies to every repos under PyTorch.

In the first case, it's perfectly ok to automatically add ciflow label.
In the second case, even if you have the mechanism to add ciflow automatically, the bot will remove the label from the PR if the author is an unknown contributor with only read permission. When this happens, the reviewer needs to review the PR, approve CI run, then add the label manually anyway. This is the manual process that I'm referring to.

So, let's be clear that adding the label automatically is doable, but only in the first case for privilege users. The mechanism to automatically add the label is independent from the change in this PR, so you can do it in a separate PR later. We need:

A labeler.yml rule like https://github.com/pytorch/pytorch/blob/main/.github/labeler.yml#L11. This works in most cases, e.g. the PR change a file and ciflow label is added
For more complex logic like using regex, we might need to update autoLabelbot at https://github.com/pytorch/test-infra/blob/main/torchci/lib/bot/autoLabelBot.ts#L26

One note is that when auto label rule is in place, having both the ciflow push trigger and pull_request triggers feel redundant, and the latter can be removed.

tianyu-l · 2025-12-07T08:23:24Z

.github/pytorch-probot.yml

@@ -0,0 +1,2 @@
+ciflow_push_tags:
+- ciflow/8gpu


what does ciflow_push_tags do?
why it's called 8gpu?

The ciflow_push_tags tells the bot which labels should map to git tags that trigger CI workflows.

It's called 8gpu following Huy's suggestion on slack https://pytorch.slack.com/archives/C0A0E8ARXPZ/p1764843858514179

It seems adding ciflow/8gpu label in a PR will automatically create a tag ciflow/8gpu/[PR_number]. I say so because I observed https://github.com/pytorch/torchtitan/releases/tag/ciflow%2F8gpu%2F2125

What is this ciflow_push_tags for? Is it for letting the bot know that the above happened and which tag to look at?

That is correct. As per my understanding, when we add PR label, it'll automatically create & push tag. Because the ciflow_push_tags config tells the bot to create & push the matching tags so that tag-based CI workflows can run.

.github/workflows/integration_test_8gpu_features.yaml

tianyu-l

Is it true that

for each PR a codeowner needs to explicitly add the ciflow/8gpu label, so that rocm test could be run?
if a codeowner doesn't do this, the PR can be merged without rocm test passing?
after CUDA test is run / finished, if we add the label, we can run rocm test without rerun the CUDA test?

.github/workflows/integration_test_8gpu_features.yaml

tianyu-l · 2025-12-09T22:01:38Z

.github/pytorch-probot.yml

@@ -0,0 +1,2 @@
+ciflow_push_tags:
+- ciflow/8gpu


It seems adding ciflow/8gpu label in a PR will automatically create a tag ciflow/8gpu/[PR_number]. I say so because I observed https://github.com/pytorch/torchtitan/releases/tag/ciflow%2F8gpu%2F2125

What is this ciflow_push_tags for? Is it for letting the bot know that the above happened and which tag to look at?

tianyu-l · 2025-12-09T22:02:13Z

.github/workflows/integration_test_8gpu_features.yaml

+    tags:
+      - ciflow/8gpu/*


What's this for -- why do we need both pull_request and tags?

As per my understanding the PR workflows and tag workflows are totally independent. Tags provides CI flow, meaning tags can be pushed to trigger CI runs on specific commits even after the PR is closed. They can also be used for versioning releases.
pull_request are used to run workflows when the PR is open.

…feature to add label automatically.

…sh tags.

…RIGGERED_8GPU_LABEL.

tianyu-l · 2025-12-11T10:40:06Z

.github/labeler.yml

+- .ci/docker/**
+- .github/workflows/**
+- scripts/**
+- tests/**
+- torchtitan/**


what are these for?

.github/workflows/integration_test_8gpu_features.yaml

tianyu-l · 2025-12-11T10:42:51Z

.github/workflows/set-matrix.yaml

+      IS_SCHEDULE:  ${{ github.event_name == 'schedule' }}
+      IS_PR:        ${{ github.event_name == 'pull_request' }}
+      HAS_8GPU_LABEL: ${{ github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ciflow/8gpu') }}
+      IS_8GPU_TAG:  ${{ startsWith(github.ref, 'refs/tags/ciflow/8gpu/') }}


why do we need this, if we already have the one above?

Implement ciflow/rocm.

0539dbe

akashveramd self-assigned this Dec 5, 2025

akashveramd requested review from fegin, tianyu-l, wconstab and wwwjn as code owners December 5, 2025 03:18

pytorch-bot bot added ciflow/rocm module: rocm labels Dec 5, 2025

meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 5, 2025

akashveramd mentioned this pull request Dec 5, 2025

Torchtitan CI gap Between ROCm & CUDA #2098

Open

10 tasks

wwwjn reviewed Dec 5, 2025

View reviewed changes

.github/pytorch-probot.yml Outdated Show resolved Hide resolved

.github/pytorch-probot.yml Show resolved Hide resolved

.github/pytorch-probot.yml Outdated Show resolved Hide resolved

huydhn reviewed Dec 5, 2025

View reviewed changes

.github/pytorch-probot.yml Outdated Show resolved Hide resolved

huydhn reviewed Dec 5, 2025

View reviewed changes

.github/pytorch-probot.yml Outdated Show resolved Hide resolved

akashveramd added 2 commits December 5, 2025 13:02

Removed tracking_issue & ciflow_tracking_issue.

19a5a15

Removed ciflow/rocm tag.

a2a55a7

tianyu-l reviewed Dec 7, 2025

View reviewed changes

Added condition to run ROCm workflow when label is added.

cc29b39

tianyu-l reviewed Dec 9, 2025

View reviewed changes

akashveramd added 6 commits December 9, 2025 20:06

Moved dynamic matrix creation to a separate reusable workflow. Added …

4485cd1

…feature to add label automatically.

Using set instead of write in set-matrix.

132a3eb

Removed trailing slash from 'ciflow/8gpu' label.

75a1d53

Refactored set-matrix.yaml. Added option to create ROCm matrix for pu…

8f8d9df

…sh tags.

Fixed indentation and Lint in set-matrix.yaml.

9417b90

Fixed indentation in set-matrix.yaml.

924444d

akashveramd added ciflow/8gpu and removed ciflow/8gpu labels Dec 11, 2025

Run ROCm workflow only when PR label added.

e2b8f12

akashveramd added ciflow/8gpu and removed ciflow/8gpu labels Dec 11, 2025

Changed the name to TRIGGERED_8GPU_LABEL. Setting default value for T…

db8cb50

…RIGGERED_8GPU_LABEL.

tianyu-l reviewed Dec 11, 2025

View reviewed changes

Implement ciflow/rocm #2114

Are you sure you want to change the base?

Implement ciflow/rocm #2114

Conversation

akashveramd commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tianyu-l commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

akashveramd commented Dec 5, 2025

Uh oh!

huydhn commented Dec 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tianyu-l left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akashveramd Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

akashveramd Dec 11, 2025 •

edited

Loading