Run the upstream HDBSCAN tests in CI

[jcrist]

This adds the HDBSCAN upstream test suite to CI.

A few complications:

• The upstream hdbscan library has seen a lapse in maintenance lately. Thus, the test suite won't run cleanly (or even import) without sklearn < 1.6. Rather than pinning the sklearn version across the upstream test jobs (which would require a new entry in dependencies.yaml and changes to the test_python_common.sh script), for now we just skip the HDBSCAN tests if that condition isn't met. The oldest deps run will run the HDBSCAN tests, but not the newest deps run.
• There are many failures in the HDBSCAN tests, most of which are memory errors. The xfail list here is stable (I've ran it 10s of times to be sure). That said, if you use pytest-randomly to shuffle the test order, you'll see a different set of failures. Not great. I figure it's still worth getting the test infra in place though before taking a look at the underlying memory issues.

Currently on top of #6994.

Fixes #6983.

[csadorf]

Merge after #6845 and #6994.

[jcrist]

Hmmm, looks like the memory failures (while consistent on my machine) are different in CI. Two tasks then:

• I'll look into marking more of them flaky to attempt to get the infra in.
• I'll start on work to open reproducible issues without cuml.accel for some. HDBSCAN cudaErrorInvalidDevice if min_cluster_size >= n_samples #6985 is one already.

[jcrist]

Looked into this more this morning. There are 3 categories of failures:

• Simple python-level failures (most of these should be quick fixes)
• Tests that always fail, and break downstream tests (there's 5 of these, at least on my machine)
• Tests that sometimes fail, and break downstream tests (this list seems impossible to nail down, it might be every test)

Given this, I think we won't be able to run these tests (at least in a required manner) until the memory bugs are fixed. A few options I see for now:

• Don't merge this PR, only try to split out the always failing tests into new issues that can be resolved without the upstream hdbscan tests or cuml.accel. Try again once those are fixed.
• Merge this PR, but remove the part adding this to CI. Gives devs the infra setup so they can run the test suite to help resolve memory issues. I'd still break out issues, but we'd have more stuff already merged in.
• Merge this PR, but ignore failures. This could be done by very very liberally marking tests as flaky, or through ignoring the pytest failure in the ci bash script.

Clearly we have some bugs in our hdbscan implementation, these we should prioritize resolving. Likewise, the python-level failures are worth resolving since they're possibly easy fixes and might affect cuml.accel users running hdbscan. The actual hdbscan upstream test suite is of pretty low quality for us though, it's not clear to me that we'd gain a ton of value running it in CI beyond hdbscan smoketests (which we could as easily write ourselves in the cuml_accel_tests/integration/ directory). Given this, I'd lean towards option 1 or 2 above.

[csadorf]

I very much prefer option 2. Let's get the infrastructure set up and start working on addressing those issues. I don't think running tests in CI that fail beyond xfails is advantageous as it just creates confusion and noise. We should add some smoke tests to cuml_accel_tests/integration/ either way.

[jcrist]

/merge