[CI] SpectralEmbedding eigensolver convergence failure in scikit-learn accel tests

[csadorf]

Summary

SpectralEmbedding fails scikit-learn common estimator checks due to RAFT eigensolver convergence failure. This failure started appearing on 2026-01-13 and affects multiple unrelated PRs.

Failing test/component: test_estimators[SpectralEmbedding()-check_estimators_*]

Failure observed in:

• https://github.com/rapidsai/cuml/actions/runs/20950779271/job/60204591125?pr=7667
• https://github.com/rapidsai/cuml/actions/runs/20952234810/job/60209553297?pr=7651
• and others

Test Details

• Test file: sklearn/tests/test_common.py
• Test names:
  • test_estimators[SpectralEmbedding()-check_estimators_fit_returns_self]
  • test_estimators[SpectralEmbedding()-check_estimators_fit_returns_self(readonly_memmap=True)]
  • test_estimators[SpectralEmbedding()-check_estimators_overwrite_params]
• Error message:

  RuntimeError: exception occurred! file=.../include/raft/linalg/detail/eig.cuh line=138: 
  eig.cuh: eigensolver couldn't converge to a solution. 
  This usually occurs when some of the features do not vary enough.
  

The error originates from raft::linalg::detail::eigDC called via cuml.manifold.SpectralEmbedding.fit when running scikit-learn's common estimator checks through cuml-accel.

Checklist

• Revert xfail check_estimators spectral embedding tests #7677

[aamijar]

I am not able to reproduce this locally with a clean environment on either gh200 or v100 with scikit-learn 1.8, 1.7, or 1.5 with CUDA Version 12.9

[betatim]

When this is solved we should undo #7677 again

[aamijar]

I was able to reproduce this problem on a L4 GPU machine.

I am using the following, which is effectively what the upstream sklearn tests in question are using:

from sklearn.datasets import make_blobs
from cuml.manifold import SpectralEmbedding

X, y = make_blobs(random_state=0, n_samples=21)
embedding = SpectralEmbedding(random_state=0)
X_embedded = embedding.fit_transform(X)

The SpectralEmbedding parameter random_state=0 causes a convergence issue, but other random_state values work fine.
On other GPU architectures such as v100, a random_state=0 value works fine. However for example a random_state=10 causes a convergence issue.

It seems that for this particular example, the convergence is sensitive to hardware and random_state, which is may be expected.
However, this dataset is example is not a realistic scenario because X already has 2 features (shape=(21,2)). And we are calling Spectral Embedding to reduce to 2 features again, which is a ill-defined scenario.

I built with the commit prior to rapidsai/cuvs#1628 and a similar thing occurs where it happens to be that random_state=0 works fine (which is what runs in CI which is why it was passing), but changing to random_state=15 would fail.
So rapidsai/cuvs#1628 simply surfaced this problem since now it uses a different solver codepath but now it happens to be that random_state=0 fails but other random_state values work fine.

[betatim]

Thanks for the investigation and the snippet. I had to modify it a bit to find a seed that breaks:

from sklearn.datasets import make_blobs
from cuml.manifold import SpectralEmbedding

X, y = make_blobs(random_state=0, n_samples=21)
for n in range(100):
  embedding = SpectralEmbedding(random_state=n)
  X_embedded = embedding.fit_transform(X)

For n=94 it finally failed :D

I had a look at the instance generator used for the common tests. For a similar check it configures n_components=1 for SpectralEmbedding. When I use SpectralEmbedding(random_state=n, n_components=1) in the above snippet and range(500) I don't see a failure. It seems to me that we could propose using n_components=1 for this check. Does this also fix the problem in your setup?

If it also solves it for you, let's make a PR to scikit-learn to propose using n_components=1 for this check. As you said, using a dataset with n_features=2 and n_components=2 can be a bit weird (though I don't know for sure that this is a "illegal" thing to ask the algorithm to do. For sure a bit unusual).

[csadorf]

It looks like these tests are flaky:

• https://github.com/rapidsai/cuml/actions/runs/21023489264/job/60483181545#step:12:1949
• https://github.com/rapidsai/cuml/actions/runs/21023489264/job/60483181583#step:12:1950

I'll introduce a PR to mark them as flaky.