[BUG] A warning (Warning: could not fill working set) poped up when fitting svm.SVC

[kliu0010]

Describe the bug
When fitting Pipeline([('scaler', StandardScaler()), ('svc', SVC(kernel="rbf", gamma="scale"))]), I got the following warning:

[W] [15:01:53.908026] Warning: could not fill working set, found only 835 elements
[W] [15:01:53.910185] Warning: could not fill working set, found only 869 elements

I was unable to find out what led to the warning above. Does this behavior have a negative influence on the training of SVC?

Steps/Code to reproduce bug

from cuml.pipeline import Pipeline
from cuml.preprocessing import StandardScaler
from cuml.svm import SVC
import numpy as np
import torch


def sample_uniformly_quasi(num, lb, ub, dtype=torch.float64):
    assert num > 0
    assert lb.ndim == 1
    assert ub.ndim == 1
    assert lb.shape[0] == ub.shape[0]
    assert np.all(lb <= ub)

    seed = np.random.randint(int(1e6))
    sobol = torch.quasirandom.SobolEngine(dimension=ub.shape[0], scramble=True, seed=seed)
    xs = sobol.draw(num, dtype=dtype).cpu().detach().numpy()
    xs = lb + (ub - lb) * xs

    return xs


dims = 2
lb = np.zeros((dims,))
ub = np.ones((dims,)) * 5.0

num = 1050
# generate random points using torch.quasirandom.SobolEngine,
# and project to the hypercube with bounds lb and ub
train_x = sample_uniformly_quasi(num=num, lb=lb, ub=ub)
train_y = np.zeros(shape=(num,), dtype=np.int64)
indices = np.random.choice(num, size=500, replace=False)
train_y[indices] = 1
train_weight = np.random.rand(train_x.shape[0])

svm_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svc', SVC(kernel="rbf", gamma="scale"))
])
svm_pipeline.fit(X=train_x, y=train_y, svc__sample_weight=train_weight)

Expected behavior
I expected that no warnings or errors would be raised.

Environment details (please complete the following information):

• Environment location: Docker
• Linux Distro/Architecture: Ubuntu 20.04 amd64
• GPU Model/Driver: RTX 3090 and driver 470.223.02
• CUDA: 11.3
• Method of cuDF & cuML install:
  • cuML was installed using pip under a conda virtual environment.
  • The results of conda list under the corresponding conda virtual environment:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
abseil-cpp                20211102.0           hd4dd3e8_0    defaults
absl-py                   1.3.0            py39h06a4308_0    defaults
aiohttp                   3.8.3            py39h5eee18b_0    defaults
aiosignal                 1.2.0              pyhd3eb1b0_0    defaults
asttokens                 2.0.5              pyhd3eb1b0_0    defaults
async-timeout             4.0.2            py39h06a4308_0    defaults
attrs                     22.1.0           py39h06a4308_0    defaults
backcall                  0.2.0              pyhd3eb1b0_0    defaults
bayesian-optimization     1.4.2                    pypi_0    pypi
blas                      1.0                         mkl    defaults
blinker                   1.4              py39h06a4308_0    defaults
botorch                   0.8.3                    pypi_0    pypi
bottleneck                1.3.5            py39h7deecbd_0    defaults
brotli                    1.0.9                h5eee18b_7    defaults
brotli-bin                1.0.9                h5eee18b_7    defaults
brotlipy                  0.7.0           py39h27cfd23_1003    defaults
bzip2                     1.0.8                h7b6447c_0    defaults
c-ares                    1.19.0               h5eee18b_0    defaults
ca-certificates           2023.12.12           h06a4308_0    defaults
cachetools                4.2.2              pyhd3eb1b0_0    defaults
certifi                   2023.11.17       py39h06a4308_0    defaults
cffi                      1.15.1           py39h5eee18b_3    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     8.1.7                    pypi_0    pypi
cloudpickle               2.2.1                    pypi_0    pypi
cma                       3.3.0                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
contourpy                 1.0.5            py39hdb19cb5_0    defaults
cryptography              38.0.4           py39h9ce1e76_0    defaults
cubinlinker-cu11          0.3.0.post1              pypi_0    pypi
cuda-python               11.8.2                   pypi_0    pypi
cudatoolkit               11.3.1               h2bc3f7f_2    defaults
cudf-cu11                 23.12.1                  pypi_0    pypi
cuml-cu11                 23.12.0                  pypi_0    pypi
cupy-cuda11x              12.3.0                   pypi_0    pypi
cycler                    0.11.0             pyhd3eb1b0_0    defaults
cython                    3.0.8                    pypi_0    pypi
dask                      2023.11.0                pypi_0    pypi
dask-cuda                 23.12.0                  pypi_0    pypi
dask-cudf-cu11            23.12.0                  pypi_0    pypi
dbus                      1.13.18              hb2f20db_0    defaults
decorator                 5.1.1              pyhd3eb1b0_0    defaults
distributed               2023.11.0                pypi_0    pypi
exceptiongroup            1.0.4            py39h06a4308_0    defaults
executing                 0.8.3              pyhd3eb1b0_0    defaults
expat                     2.4.9                h6a678d5_0    defaults
fastrlock                 0.8.2                    pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
fftw                      3.3.10          nompi_hf0379b8_106    conda-forge
flit-core                 3.6.0              pyhd3eb1b0_0    defaults
fontconfig                2.14.1               h52c9d5c_1    defaults
fonttools                 4.25.0             pyhd3eb1b0_0    defaults
freetype                  2.12.1               h4a9f257_0    defaults
frozenlist                1.3.3            py39h5eee18b_0    defaults
fsspec                    2023.12.2                pypi_0    pypi
giflib                    5.2.1                h5eee18b_1    defaults
glib                      2.69.1               he621ea3_2    defaults
gmp                       6.2.1                h295c915_3    defaults
gnutls                    3.6.15               he1e5248_0    defaults
google-auth               2.6.0              pyhd3eb1b0_0    defaults
google-auth-oauthlib      0.4.4              pyhd3eb1b0_0    defaults
gpytorch                  1.9.1                    pypi_0    pypi
grpc-cpp                  1.48.2               h5bf31a4_0    defaults
grpcio                    1.48.2           py39h5bf31a4_0    defaults
gst-plugins-base          1.14.1               h6a678d5_1    defaults
gstreamer                 1.14.1               h5eee18b_1    defaults
gym                       0.26.2                   pypi_0    pypi
gym-notices               0.0.8                    pypi_0    pypi
icu                       58.2                 he6710b0_3    defaults
idna                      3.4              py39h06a4308_0    defaults
imageio                   2.26.0           py39h06a4308_0    defaults
importlib-metadata        6.1.0                    pypi_0    pypi
importlib_resources       5.2.0              pyhd3eb1b0_1    defaults
intel-openmp              2021.4.0          h06a4308_3561    defaults
ipython                   8.15.0           py39h06a4308_0    defaults
jedi                      0.18.1           py39h06a4308_1    defaults
jinja2                    3.1.3                    pypi_0    pypi
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h7f8727e_0    defaults
kiwisolver                1.4.4            py39h6a678d5_0    defaults
krb5                      1.19.4               h568e23c_0    defaults
lame                      3.100                h7b6447c_0    defaults
lcms2                     2.12                 h3be6417_0    defaults
ld_impl_linux-64          2.38                 h1181459_1    defaults
lerc                      3.0                  h295c915_0    defaults
libbrotlicommon           1.0.9                h5eee18b_7    defaults
libbrotlidec              1.0.9                h5eee18b_7    defaults
libbrotlienc              1.0.9                h5eee18b_7    defaults
libclang                  10.0.1          default_hb85057a_2    defaults
libdeflate                1.8                  h7f8727e_5    defaults
libedit                   3.1.20221030         h5eee18b_0    defaults
libevent                  2.1.12               h8f2d780_0    defaults
libffi                    3.4.2                h6a678d5_6    defaults
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libiconv                  1.16                 h7f8727e_2    defaults
libidn2                   2.3.2                h7f8727e_0    defaults
libllvm10                 10.0.1               hbcb73fb_5    defaults
libllvm11                 11.1.0               hf817b99_2    conda-forge
libpng                    1.6.37               hbc83047_0    defaults
libpq                     12.9                 h16c4e8d_3    defaults
libprotobuf               3.20.3               he621ea3_0    defaults
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libtasn1                  4.16.0               h27cfd23_0    defaults
libtiff                   4.5.0                h6a678d5_1    defaults
libunistring              0.9.10               h27cfd23_0    defaults
libuuid                   1.41.5               h5eee18b_0    defaults
libwebp                   1.2.4                h11a3e52_0    defaults
libwebp-base              1.2.4                h5eee18b_0    defaults
libxcb                    1.15                 h7f8727e_0    defaults
libxkbcommon              1.0.1                hfa300c1_0    defaults
libxml2                   2.9.14               h74e7548_0    defaults
libxslt                   1.1.35               h4e12654_0    defaults
line_profiler             4.1.1            py39hdb19cb5_0    defaults
linear-operator           0.3.0                    pypi_0    pypi
llvm-openmp               14.0.6               h9e868ea_0    defaults
llvmlite                  0.40.1                   pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
lz4-c                     1.9.4                h6a678d5_0    defaults
markdown                  3.4.1            py39h06a4308_0    defaults
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.1            py39h7f8727e_0    defaults
matplotlib                3.7.1            py39h06a4308_1    defaults
matplotlib-base           3.7.1            py39h417a72b_1    defaults
matplotlib-inline         0.1.6            py39h06a4308_0    defaults
mdurl                     0.1.2                    pypi_0    pypi
mkl                       2021.4.0           h06a4308_640    defaults
mkl-service               2.4.0            py39h7f8727e_0    defaults
mkl_fft                   1.3.1            py39hd3c417c_0    defaults
mkl_random                1.2.2            py39h51133e4_0    defaults
msgpack                   1.0.7                    pypi_0    pypi
multidict                 6.0.2            py39h5eee18b_0    defaults
multipledispatch          0.6.0                    pypi_0    pypi
munkres                   1.1.4                      py_0    defaults
ncurses                   6.4                  h6a678d5_0    defaults
nettle                    3.7.3                hbbd107a_1    defaults
nevergrad                 0.6.0                    pypi_0    pypi
nspr                      4.33                 h295c915_0    defaults
nss                       3.74                 h0370c37_0    defaults
numba                     0.57.1                   pypi_0    pypi
numexpr                   2.8.4            py39he184ba9_0    defaults
numpy                     1.23.5           py39h14f4228_0    defaults
numpy-base                1.23.5           py39h31eccc5_0    defaults
nvtx                      0.2.8                    pypi_0    pypi
oauthlib                  3.2.2            py39h06a4308_0    defaults
openh264                  2.1.1                h4ff587b_0    defaults
openssl                   1.1.1w               h7f8727e_0    defaults
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 23.0             py39h06a4308_0    defaults
pandas                    1.5.3            py39h417a72b_0    defaults
parso                     0.8.3              pyhd3eb1b0_0    defaults
partd                     1.4.1                    pypi_0    pypi
pcre                      8.45                 h295c915_0    defaults
pexpect                   4.8.0              pyhd3eb1b0_3    defaults
pickleshare               0.7.5           pyhd3eb1b0_1003    defaults
pillow                    9.3.0            py39h6a678d5_2    defaults
pip                       22.3.1           py39h06a4308_0    defaults
ply                       3.11             py39h06a4308_0    defaults
prompt-toolkit            3.0.43           py39h06a4308_0    defaults
protobuf                  4.25.2                   pypi_0    pypi
psutil                    5.9.7                    pypi_0    pypi
ptxcompiler-cu11          0.7.0.post1              pypi_0    pypi
ptyprocess                0.7.0              pyhd3eb1b0_2    defaults
pure_eval                 0.2.2              pyhd3eb1b0_0    defaults
pyarrow                   14.0.2                   pypi_0    pypi
pyasn1                    0.4.8              pyhd3eb1b0_0    defaults
pyasn1-modules            0.2.8                      py_0    defaults
pybox2d                   2.3.10           py39h5a03fae_4    conda-forge
pycparser                 2.21               pyhd3eb1b0_0    defaults
pygame                    2.5.1                    pypi_0    pypi
pygments                  2.15.1           py39h06a4308_1    defaults
pyjwt                     2.4.0            py39h06a4308_0    defaults
pylibraft-cu11            23.12.0                  pypi_0    pypi
pynvml                    11.4.1                   pypi_0    pypi
pyopenssl                 22.0.0             pyhd3eb1b0_0    defaults
pyparsing                 3.0.9            py39h06a4308_0    defaults
pyqt                      5.15.7           py39h6a678d5_1    defaults
pyqt5-sip                 12.11.0          py39h6a678d5_1    defaults
pyro-api                  0.1.2                    pypi_0    pypi
pyro-ppl                  1.8.4                    pypi_0    pypi
pysocks                   1.7.1            py39h06a4308_0    defaults
python                    3.9.16               h7a1cb2a_0    defaults
python-dateutil           2.8.2              pyhd3eb1b0_0    defaults
python_abi                3.9                      2_cp39    conda-forge
pytorch                   1.12.1          py3.9_cuda11.3_cudnn8.3.2_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2022.7           py39h06a4308_0    defaults
pyyaml                    6.0.1                    pypi_0    pypi
qt-main                   5.15.2               h327a75a_7    defaults
qt-webengine              5.15.9               hd2b0992_4    defaults
qtwebkit                  5.212                h4eab89a_4    defaults
raft-dask-cu11            23.12.0                  pypi_0    pypi
rapids-dask-dependency    23.12.1                  pypi_0    pypi
re2                       2022.04.01           h295c915_0    defaults
readline                  8.2                  h5eee18b_0    defaults
requests                  2.28.1           py39h06a4308_0    defaults
requests-oauthlib         1.3.0                      py_0    defaults
rich                      13.7.0                   pypi_0    pypi
rmm-cu11                  23.12.0                  pypi_0    pypi
rsa                       4.7.2              pyhd3eb1b0_1    defaults
scikit-learn              1.2.1            py39h6a678d5_0    defaults
scipy                     1.10.1                   pypi_0    pypi
setuptools                65.6.3           py39h06a4308_0    defaults
sip                       6.6.2            py39h6a678d5_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
sortedcontainers          2.4.0                    pypi_0    pypi
sqlite                    3.40.1               h5082296_0    defaults
stack_data                0.2.0              pyhd3eb1b0_0    defaults
tblib                     3.0.0                    pypi_0    pypi
tensorboard               2.11.0           py39h06a4308_0    defaults
tensorboard-data-server   0.6.1            py39h52d8a92_0    defaults
tensorboard-plugin-wit    1.8.1            py39h06a4308_0    defaults
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h1ccaba5_0    defaults
toml                      0.10.2             pyhd3eb1b0_0    defaults
toolz                     0.12.0                   pypi_0    pypi
torchaudio                0.12.1               py39_cu113    pytorch
torchvision               0.13.1               py39_cu113    pytorch
tornado                   6.2              py39h5eee18b_0    defaults
tqdm                      4.65.0                   pypi_0    pypi
traitlets                 5.7.1            py39h06a4308_0    defaults
treelite                  3.9.1                    pypi_0    pypi
treelite-runtime          3.9.1                    pypi_0    pypi
tslearn                   0.5.3.2          py39h389d5f1_0    conda-forge
typing_extensions         4.4.0            py39h06a4308_0    defaults
tzdata                    2022g                h04d1e81_0    defaults
ucx-py-cu11               0.35.0                   pypi_0    pypi
urllib3                   1.26.14          py39h06a4308_0    defaults
wcwidth                   0.2.5              pyhd3eb1b0_0    defaults
werkzeug                  2.2.3            py39h06a4308_0    defaults
wheel                     0.37.1             pyhd3eb1b0_0    defaults
xz                        5.2.10               h5eee18b_1    defaults
yarl                      1.8.1            py39h5eee18b_0    defaults
zict                      3.0.0                    pypi_0    pypi
zipp                      3.15.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0    defaults
zstd                      1.5.2                ha4553b6_0    defaults

[dantegd]

Thanks for the issue @OpsTreadstone, if I'm not mistaken this shouldn't affect things as long as the results you're seeing are within expectations, but @tfeher or @achirkin would be able to respond this better than myself.

[xf67]

Same issue here, with Ubuntu20.04, amd64, RTX3090, Driver version 555.42.02, CUDA version 12.5.

However, the accuracy is not affected or slightly affected in my experiment.

If you wanna suppress the warning, you can use code like below:

import contextlib
import io
from cuml.svm import LinearSVC
svm = LinearSVC()
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
    svm.fit(X_train, y_train)
y_pred = svm.predict(X_test)

Violent but effective approach.

[aamijar]

Hi, you can set the log level like this, svm = LinearSVC(verbose=0)

[csadorf]

Hi, you can set the log level like this, svm = LinearSVC(verbose=0)

Should we still adjust the default verbosity?

[aamijar]

Warn level as default should be okay since warning messages could be useful to know?

[csadorf]

Yes, that's fair. I was under the impression that we saw an excessive number of warning messages. We should make sure that there is only one warning per fit() call. The number of warnings should not scale with input data size.

We should also make sure that standard methods to control verbosity are respected. This might be related to #6394 .

[aamijar]

Is it possible to check whether a previous log message was shown and then suppress the following ones?

[aamijar]

@csadorf Do you think a pattern like this would be okay to stop the excessive logging for the SVC in particular?

if (!warning_logged) {
  CUML_LOG_WARN(
        "Warning: could not fill working set, found only %d"
        " elements",
        n_already_selected);
  warning_logged = True;
}