[BACKEND] combineRedundantWaitOps should not combine across loops/branches #7593
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AlexAUT
force-pushed
the
fixCombineRedundantWaitOps
branch
from
8ef6ff8 to
32a4271
Compare
antiagainst
approved these changes
Jul 22, 2025
Collaborator
|
LGTM; also adding @pawelszczerbuk to take a look. |
pawelszczerbuk
approved these changes
Jul 22, 2025
Contributor
pawelszczerbuk
left a comment
pawelszczerbuk
left a comment
There was a problem hiding this comment.
Makes sense to me! Thanks!
AlexAUT
added a commit
to ROCm/triton
that referenced
this pull request
Jul 30, 2025
…nches (triton-lang#7593) `combineRedundantWaitOps` did skip over branches/loops, so if we end up with something like: ```mlir ttg.async_wait scf.for .... scf.yield ttg.async_wait ``` we merge the async_waits in the prologue and epilogue because we do not find a `ttg.commit_group` in between. This PR stops the forward search if we encounter a branch/loop. I can also walk through all successor blocks if we think this is worth the effort. This problem was not triggered before because the `ttg.async_wait` was scheduled in the same stage as its user(s) so we ended up with no `ttg.async_wait` in the prologue or there was another prefetch after it in the prologue. Since triton-lang#7458 we might place the `ttg.async_wait` in the previous stage compared to its user(s) so we might end up with the problematic IR.
AlexAUT
added a commit
to ROCm/triton
that referenced
this pull request
Jul 31, 2025
…nches (triton-lang#7593) `combineRedundantWaitOps` did skip over branches/loops, so if we end up with something like: ```mlir ttg.async_wait scf.for .... scf.yield ttg.async_wait ``` we merge the async_waits in the prologue and epilogue because we do not find a `ttg.commit_group` in between. This PR stops the forward search if we encounter a branch/loop. I can also walk through all successor blocks if we think this is worth the effort. This problem was not triggered before because the `ttg.async_wait` was scheduled in the same stage as its user(s) so we ended up with no `ttg.async_wait` in the prologue or there was another prefetch after it in the prologue. Since triton-lang#7458 we might place the `ttg.async_wait` in the previous stage compared to its user(s) so we might end up with the problematic IR.
jataylo
pushed a commit
to ROCm/triton
that referenced
this pull request
Aug 1, 2025
* [AMD] Avoid async load to pipeline for less than 32bit load (triton-lang#7250) We can only use AsyncCopy if the final load width can be >= 4 bytes. `triton::canBeConvertedToAsyncLoad` checks that the vecSize of the source is large enough. Additionally we need to ensure the register to shared layout (blocked+shared) does have enough contiguous elements since we cannot scatter into LDS. Before this PR we will abort compilation instead of falling back to pipelining through registers. * [AMD] Pipeline small tensors w/ registers only on GFX950 (triton-lang#7171) Fixes a perf regression on gfx942 but preserves functionality for gfx950 (and above). * Reland "[AMD] Optimize reduction with v_permlane intrinsics in GFX950" (triton-lang#7321) triton-lang#7291 fixed the LLVM issue that caused correctness problems. Now we can reland this patch. * [Pipeliner] Expose core pipeliner utilities to reuse in AMD backend (triton-lang#7222) This PR exposes (via header) core pipelining utilities/helpers: ```c++ bool hasGpuBarriers(scf::ForOp forOp); bool isSafeToPipeline(scf::ForOp forOp); llvm::MapVector<Operation *, std::pair<int, Operation *>> loadOpsToIndirectionLevel(scf::ForOp forOp, bool pipelineWithoutDot, triton::ModuleAxisInfoAnalysis &axisInfoAnalysis, int numStages, bool filterSmall = true); void scheduleDistanceOneDependencies(scf::ForOp forOp, CoarseSchedule &schedule); void scheduleRemainingToLastStage(scf::ForOp forOp, CoarseSchedule &schedule, CoarseSchedule::Cluster afterPrologue); ``` They are directly useable by AMD's pipeliner. Note, this is basically NFC for AMD because AMD's pipeliner simply had copy-paste of the same functions from ~last year. Small API changes: 1. On NV we do not pipeline small loads (vec width < 32b). On AMD we do. The choice is made inside `isPipeliningBeneficial` inside `loadOpsToIndirectionLevel`. To support AMD I have added a flag `filterSmall`. 2. On AMD the load `use`s (computed as a matter of course in `loadOpsToIndirectionLevel`) are used (no pun intended) whereas on NV they are not. To support AMD I keep those `use`s in the `llvm::MapVector<Operation *, std::pair<int, Operation *>>` return from `loadOpsToIndirectionLevel`. These two small changes are the only "non-NFC" changes. * [AMD] Retire local prefetch schedule hint variant (triton-lang#7395) This variant was from some prior experiments. We have a better way to implement later. * [AMD] Retire TritonAMDGPU_OpIdxAttr and TritonAMDGPU_InstCounter (triton-lang#7476) triton-lang#7395 retired the local prefetch schedule variant. This made `TritonAMDGPU_OpIdxAttr` and `TritonAMDGPU_InstCounter` unused which are removed by this PR. * [AMD][NFC] Split createAndSchedule* in stream pipeliner(triton-lang#7514) Splits `createAndScheduleAsyncCopy` and `createAndScheduleStreamCopy` to make it reusable if we want to schedule the ops differently in a future PR. * [AMD] Refactor StreamPipeliner to use more common functions (triton-lang#7526) Further refactoring of Streampipeliner.cpp to use more common pipeliner functionality: `triton::createAllocation`, `triton::createSingleBufferView`, `triton::replaceWithSharedLoad` and a bit of general cleanup. Overall NFC except: - The order of LocalDealloc is reversed now - The memdesc of the subview additionally includes the allocSize Also we had no lit test checking that the LocalLoad consumes the AsyncToken so I adjusted one to include the check. * [AMD] NFC: Refactor stream pipeliner to better encapsulate functions (triton-lang#7540) Mostly moves code around to reduce the dependencies between functions and further splits up functions doing more than one thing (`createAndSchedule*`,` preprocessAndBuildSchedule`). This will also allow us to use more common pipeliner functionality in a future PR, e.g. `createAsyncCopy`. * [FA] Set vecSize=nonKDim for V shared layout to avoid bank conflicts I'll submit a PR upstream later. * [GEMM] Add combine dot_scaled and addF * [AMD][NFC] Consolidate initialization in initSchedule for pipeliner (triton-lang#7556) Moves all initializations of stages to `initSchedule`. Missed this one in the last PRs. * [AMD] NFC: Drop version minor for AMD MFMA layout (triton-lang#7285) AMD's MFMA layout does not need version minor information like NVIDIA. It always defaults to 0 in the current codebase. The PR drops version minor and change to a single `version` parameter for MFMA layout. * [AMD] Add tilesPerWarp parameter to mfma layout (triton-lang#7283) This PR introduces the tilesPerWarp parameter to the MFMA layout. Previously, the MFMA layout assumed that each warp within a CTA tile computed a single MFMA tile. When the tensor was larger than a single CTA tile, these tiles were repeated across the tensor. In this setup, the output tiles computed by each wave were strided by the number of warps per CTA in both row and column dimensions. For instance, with 16 MFMA tiles and warpsPerCTA = [2, 2], the distribution of warps across the MFMA tiles looked like: w0 w1 w0 w1 w2 w3 w2 w3 w0 w1 w0 w1 w2 w3 w2 w3 The new tilesPerWarp parameter allows each warp to compute contiguous MFMA tiles in the row and/or column dimensions. Using the same example with tilesPerWarp = [2, 2], the layout becomes: w0 w0 w1 w1 w0 w0 w1 w1 w2 w2 w3 w3 w2 w2 w3 w3 While this is a general enhancement, the main motivation for introducing this parameter is to improve memory access efficiency for scale tensors in scaled dot operations. Specific patterns and use cases will be implemented in follow-up PRs. --------- Co-authored-by: Ognjen Plavsic <[email protected]> Co-authored-by: Lei Zhang <[email protected]> * [AMD] Add support for pingpong GEMM using async_copy * [BACKEND] combineRedundantWaitOps should not combine across loops/branches (triton-lang#7593) `combineRedundantWaitOps` did skip over branches/loops, so if we end up with something like: ```mlir ttg.async_wait scf.for .... scf.yield ttg.async_wait ``` we merge the async_waits in the prologue and epilogue because we do not find a `ttg.commit_group` in between. This PR stops the forward search if we encounter a branch/loop. I can also walk through all successor blocks if we think this is worth the effort. This problem was not triggered before because the `ttg.async_wait` was scheduled in the same stage as its user(s) so we ended up with no `ttg.async_wait` in the prologue or there was another prefetch after it in the prologue. Since triton-lang#7458 we might place the `ttg.async_wait` in the previous stage compared to its user(s) so we might end up with the problematic IR. * [AMD][NFC] Group scheduling functions in StreamPipeliner (triton-lang#7607) NFC: Groups all scheduling related function to a namespace to prepare for additional scheduling variants. * [AMD] Add pingpong transformation for chained dot schedule (triton-lang#7638) Adds support to enable pingpong for loops scheduled with the new `ChainedDotSchedule` introduced by triton-lang#7601. The schedule already places the ops in the correct order so we just have to insert the sync ops to ensure proper pingpong'ing. * [AMD] Fix pingpong ChainedDot for empty second memory cluster (triton-lang#7694) triton-lang#7638 introduced a null pointer access (during review adjustments) if the second memory cluster is empty or if there are no memory clusters at all. Added a lit test to catch it and revert to the old logic. * [AMD] Remove bypass permute optimization for AsyncCopy (triton-lang#7704) We can only bypass ds_bpermute to apply the swizzling if lanes loading the same row read a contiguous chunk of memory from HBM, which we cannot infer when lowering to LLVM. The current selection does only check if the elements for each lane are contiguous which is not strict enough. * [AMD] Add ChainedDotSchedule to StreamPipeliner (triton-lang#7601) Adds a new scheduling variant which kicks in for loop which have 2 chained dots and `num_stages==4`. It places the two dots in consecutive stages so we can interleave operations using the result of the first dot with both dots in the loop, a pseudo example IR: ``` %1 = tt.dot ... %2 = arith.addf %1, %arg1 %3 = arith.subf %2, %arg2 %4 = tt.dot %X, %Y, %3 ``` Which could result in the following pseudo schedule (ignoring mem ops) to interleave with both dots: ``` stage N, Cluster0: [%1 = tt.dot, %3 = arith.subf] stage N+1, Cluster1: [%4 = tt.dot, %2 = arith.addf] ``` As a first step the schedule splits the op chain between dot1 and dot2 when it encounters an operation which has more than 2 users. This aims to avoid adding too many loop carried dependencies but does not guarantee a good work balance between the two clusters. In future PRs we might make this more sophisticated. * [AMD] Add scale preshuffling and opSel implementation (triton-lang#7603) This PR implements test kernel for efficient scale packing for CDNA4 arch as well as opSel for scaled MFMA instructions. Scaled MFMA instructions expect scale operands as 32-bit values, even though each individual scale is only 8 bits. To reduce register usage, we pack 4 scales into a single 32-bit value and use the opSel field to select the appropriate byte during execution. Packing is done along the K dimension first. if there aren’t enough values in K, we continue along the non-K dimension. --------- Co-authored-by: Ognjen Plavsic <[email protected]> * [AMD] Enable Pingpong by default on gfx950 arch (triton-lang#7697) List of enabling conditions - FP/BF16 GEMM with M,N>64 tilesize when num_stages=3 and num_warps=8 - GEMM using `dot_scaled` with M=N=256 tile size when num_stages=2 and num_warps=8 - FA with num_stages=4 Only with using async_copy. * [Backend] Bump to llvm/llvm-project@570885128351 (triton-lang#7291) This picks up a bug fix for AMDGPU v_permlane_swap: llvm/llvm-project#144423 Without this fix, the v_permlane_swap is wrongly sunk. Along the way we need to fix API changes: Add header file for the class IRBuilder Add missing default parameter in convertFuncOpToLLVMFuncOp --------- Co-authored-by: Maksim Levental <[email protected]> Co-authored-by: Yi Qian <[email protected]> Co-authored-by: Lei Zhang <[email protected]> Co-authored-by: Lixun Zhang <[email protected]> Co-authored-by: Jungwook Park <[email protected]> Co-authored-by: Pengzhan Zhao <[email protected]> Co-authored-by: plognjen <[email protected]> Co-authored-by: Ognjen Plavsic <[email protected]> Co-authored-by: Zeng Wu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
combineRedundantWaitOpsdid skip over branches/loops, so if we end up with something like:we merge the async_waits in the prologue and epilogue because we do not find a
ttg.commit_groupin between. This PR stops the forward search if we encounter a branch/loop. I can also walk through all successor blocks if we think this is worth the effort.This problem was not triggered before because the
ttg.async_waitwas scheduled in the same stage as its user(s) so we ended up with nottg.async_waitin the prologue or there was another prefetch after it in the prologue.Since #7458 we might place the
ttg.async_waitin the previous stage compared to its user(s) so we might end up with the problematic IR.