Skip to content

[BUG] Nightly CI issue: kerneldensity cupy fusekernels fail hypothesis flaky test in CUDA 12.0.1 job with cupy compilation error. #6173

@dantegd

Description

@dantegd

Nightly jobs show the following error:

E               hypothesis.errors.FlakyReplay: Inconsistent results from replaying a test case!
E                 last: INTERESTING from CompileException at /opt/conda/envs/test/lib/python3.11/site-packages/cupy/cuda/compiler.py:711
E                   context: NVRTCError at cupy_backends/cuda/libs/nvrtc.pyx:53
E                       context: TypeError at cupy/_core/fusion.pyx:554
E                 this: INTERESTING from CompileException at /opt/conda/envs/test/lib/python3.11/site-packages/cupy/cuda/compiler.py:711
E                   context: NVRTCError at cupy_backends/cuda/libs/nvrtc.pyx:53

Digging deeper this is coming from CuPy:

E           cupy.cuda.compiler.CompileException: /tmp/tmpueuqs7aw/67091474e9b3014299476429db97e8d39a66649f.cubin.cu(5): catastrophic error: cannot open source file "cooperative_groups.h"

Using the same containers outside of CI this does not reproduce currently.

Metadata

Metadata

Assignees

Labels

? - Needs TriageNeed team to review and classifybugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions