[Issue]: MIOpen test failures observed against rocm-systems commit

[idass1990]

Problem Description

the following commit on rocm-systems is causing MIOpen test failures on linux (gfx94X):
ROCm/rocm-systems@840301e

Failing workflow run on breaking change (840301e): https://github.com/ROCm/TheRock/actions/runs/20347027429
Passing build/test on the commit just prior (b106e6f): https://github.com/ROCm/TheRock/actions/runs/20347312543

Operating System

Linux

CPU

AMD EPYC 9575F 64-Core

GPU

gfx94X

ROCm Version

7.11

ROCm Component

MIOpen, clr

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[ScottTodd]

Sample error logs:

[       OK ] Full/GPU_SoftMarginLossForward_FP32.Test/28 (0 ms)
[ RUN      ] Full/GPU_SoftMarginLossForward_FP32.Test/32
HIPRTC compile ROCm include path argument: -I/__w/TheRock/TheRock/build/include
/__w/TheRock/TheRock/rocm-libraries/projects/miopen/test/gtest/main_hip.cpp:16: Failure
Expected equality of these values:
  hipError
    Which is: 709
  hipSuccess
    Which is: 0
 hipGetLastError returned error code 709 after test Full/GPU_SoftMarginLossForward_FP32.Test/32. Error string: context is destroyed

[  FAILED  ] Full/GPU_SoftMarginLossForward_FP32.Test/32, where GetParam() = dims:32x80x870 strides:69600x870x1 reduction_mode:2 (318 ms)
[ RUN      ] Full/GPU_SoftMarginLossForward_FP32.Test/36
/__w/TheRock/TheRock/rocm-libraries/projects/miopen/test/gtest/main_hip.cpp:16: Failure
Expected equality of these values:
  hipError
    Which is: 709
  hipSuccess
    Which is: 0
 hipGetLastError returned error code 709 after test Full/GPU_SoftMarginLossForward_FP32.Test/36. Error string: context is destroyed

[  FAILED  ] Full/GPU_SoftMarginLossForward_FP32.Test/36, where GetParam() = dims:2x3x160x160 strides:6528000x2176000x13600x85 reduction_mode:2 (107 ms)
[ RUN      ] Full/GPU_SoftMarginLossForward_FP32.Test/40
/__w/TheRock/TheRock/rocm-libraries/projects/miopen/test/gtest/main_hip.cpp:16: Failure
Expected equality of these values:
  hipError
    Which is: 709
  hipSuccess
    Which is: 0
 hipGetLastError returned error code 709 after test Full/GPU_SoftMarginLossForward_FP32.Test/40. Error string: context is destroyed

[jayhawk-commits]

CC'ing @BradPepersAMD and @DarylHawkinsAMD

[jayhawk-commits]

The PR that reverts the clr commit has been included in the latest rocm-systems submodule bump.

[NolanHannaAMD]

For visibility, this issue is being investigated. We have a potential fix that we are working on testing. If testing is successful then we will try to get it in to unblock the rocm-systems change.