[BUG] n_neighbors should be smaller than the graph degree computed by nn descent

[thorstenwagner]

I updated to the latest cuml (from 23.12). I'm fitting a umap to dataset with 32 features and 400k samples.

With 23.12 I did that with n_neighbors=200 and n_components=2 and it worked. With the latest version (24.08) I get:

Traceback (most recent call last):
  File "/mnt/data/twagner/Projects/TomoTwin/results/test_runs/test.py", line 11, in <module>
    reducer.fit(np.random.randn(52000,32))
  File "/opt/user_software/miniconda3_envs/tomotwin2/lib/python3.11/site-packages/cuml/internals/api_decorators.py", line 188, in wrapper
    ret = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/user_software/miniconda3_envs/tomotwin2/lib/python3.11/site-packages/cuml/internals/api_decorators.py", line 393, in dispatch
    return self.dispatch_func(func_name, gpu_func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/user_software/miniconda3_envs/tomotwin2/lib/python3.11/site-packages/cuml/internals/api_decorators.py", line 190, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "base.pyx", line 687, in cuml.internals.base.UniversalBase.dispatch_func
  File "umap.pyx", line 668, in cuml.manifold.umap.UMAP.fit
RuntimeError: RAFT failure at file=/opt/conda/conda-bld/work/cpp/src/umap/knn_graph/algo.cuh line=115: n_neighbors should be smaller than the graph degree computed by nn descent
Obtained 25 stack frames

The magic n_neighbors number when it starts working is 64, which seems to be the default according this documentation: https://docs.rapids.ai/api/cuvs/stable/cpp_api/neighbors_nn_descent/

Here is a script to reproduce the issue:

import cuml
import numpy as np
reducer = cuml.UMAP(
    n_neighbors=200,
    n_components=2,
    n_epochs=None,  # means automatic selection
    min_dist=0.0,
    random_state=19,
    metric="euclidean"
)
reducer.fit(np.random.randn(400000,32))
print("Done")

Interestingly, when I reduce the number of samples from 400k to 50k it also works.

Any ideas what I'm doing wrong?

[daniyalaliev]

+, similar problem

[beckernick]

Thanks for reporting. Would you be able to give this a try with the stable v24.10 release that just came out? I'm not able to reproduce it in the current release (though I saw it in 24.08).

import cuml
import numpy as np
reducer = cuml.UMAP(
    n_neighbors=200,
    n_components=2,
    n_epochs=None,  # means automatic selection
    min_dist=0.0,
    random_state=19,
    metric="euclidean"
)
reducer.fit(np.random.randn(400000,32))
print("Done")
Done

[beckernick]

Closing, but please feel free to re-open if this reproduces in v24.10+

[aimgo]

This appears to remain an issue in not only v24.10, but also v24.12.

    # ~ Create and fit CUML UMAP
    cuml_umap = UMAP(
        n_components=2,
        n_neighbors = 100, 
        init="spectral", 
        target_weight=0.25, 
        min_dist=0.00001,
        n_epochs=1000
    ) 
    
    # ~ Fit 
    # ~ Transform
    reduced_embeddings = cuml_umap.fit_transform(X_gpu, Y_gpu) 

Used with ~ 1 million samples of dimension 784 resulted in

File ~/.local/lib/python3.12/site-packages/cuml/internals/api_decorators.py:393, in enable_device_interop.<locals>.dispatch(self, *args, **kwargs)
    391 if hasattr(self, "dispatch_func"):
    392     func_name = gpu_func.__name__
--> 393     return self.dispatch_func(func_name, gpu_func, *args, **kwargs)
    394 else:
    395     return gpu_func(self, *args, **kwargs)

File ~/.local/lib/python3.12/site-packages/cuml/internals/api_decorators.py:190, in _make_decorator_function.<locals>.decorator_function.<locals>.decorator_closure.<locals>.wrapper(*args, **kwargs)
    188         ret = func(*args, **kwargs)
    189     else:
--> 190         return func(*args, **kwargs)
    192 return cm.process_return(ret)

File base.pyx:689, in cuml.internals.base.UniversalBase.dispatch_func()

File umap.pyx:678, in cuml.manifold.umap.UMAP.fit()

RuntimeError: RAFT failure at file=/__w/cuml/cuml/cpp/src/umap/knn_graph/algo.cuh line=111: n_neighbors should be smaller than the graph degree computed by nn descent

for both versions.

[beckernick]

@cjnolet , do you expect this is as straighforward as adding a graph_degree = max(graph_degree, n_neighbors) for this codepath?

[saucechef]

While the issue still persists in the current nightly build, you can at least get it to run for larger n_neighbors by switching to brute-force KNN, circumventing the branch where the error occurs.

umap = UMAP(n_components=3,
            n_neighbors=100,
            build_algo="brute_force_knn")

Of course, you trade in the performance benefits of ANN and you also cannot use data_on_host, but it's still much faster than UMAP on CPU in my use cases.

[cjnolet]

@cjnolet , do you expect this is as straighforward as adding a graph_degree = max(graph_degree, n_neighbors) for this codepath?

Yes I do believe this would be the proper way around this.