[Epic] Improve sklearn compatibility

[csadorf]

Improve cuML's sklearn compatibility through systematic testing and validation of estimators.

Impact

• Better integration with sklearn ecosystem and meta-estimators
• Reduced limitations for cuml.accel
• Prevention of compatibility regressions
• Improved documentation of compatibility status

Objectives

O1: Establish Testing Framework

Establish check_estimator testing for all cuML estimators, even with expected failures.

• Make use of check_estimator and other standard sklearn compatibility checks #6432

O2: Address High-Impact Issues

• Identify and prioritize prevalent compatibility issues
• Resolve high-impact issues

O3: Prevent Regressions

• Make check_estimator tests opt-out rather than opt-in

Non-Goals

• Perfect compatibility: Only address high-impact issues
• Immediate resolution of all gaps: Defer low-impact issues
• Testing all sklearn versions: Single version testing is sufficient
• Exhaustive documentation: Issue-based documentation is sufficient for most cases

Timeline

• 25.10: Establish testing infrastructure and address high-impact issues
• Ongoing: Address high- and medium-impact issues

Risks and Mitigations

• Scope creep: Prioritize issues rather than fixing everything
• Maintenance burden: Keep test infrastructure lean
• False positives: Evaluate impact before adjusting API/implementation

Success Criteria

• All cuML estimators tested with check_estimator
• High-impact compatibility issues identified and resolved

[betatim]

The basic infrastructure now exists and runs. Hopefully soon a PR to scikit-learn will be merged that will let us notice xfailed checks that start passing (strict mode).

Taking a quick look at which tests have been xfailed and for which estimators. Here is the top ten, sorted by how many estimators fail a check:

1. check_estimator_tags_renamed: 25 estimators
2. check_no_attributes_set_in_init: 25 estimators
3. check_do_not_raise_errors_in_init_or_set_params: 25 estimators
4. check_complex_data: 25 estimators
5. check_dtype_object: 25 estimators
6. check_estimators_nan_inf: 25 estimators
7. check_parameters_default_constructible: 25 estimators
8. check_estimator_sparse_tag: 23 estimators
9. check_estimators_empty_data_messages: 23 estimators
10. check_dont_overwrite_parameters: 22 estimators

The top 7 could be in any order really. One group that probably has the same root cause is check_no_attributes_set_in_init, check_do_not_raise_errors_in_init_or_set_params and check_dont_overwrite_parameters - they are all related to setting and checking of hyper-parameters. Getting this right makes it more likely that our estimators work well inside hyperparameter optimisation tools, as well as being more conformant as a goal in of itself.

check_estimator_tags_renamed checks that we support the new tags infrastructure, which we don't. This is probably not too hard to do either and will help improve introspection. For example some tests are only applied to estimators with a particular tag. Related to the tags infrastructure is check_estimator_sparse_tag which checks that estimators that claim to support sparse input actually do and vice versa

check_estimators_nan_inf is about raising an exception if the input to fit, predict contains Nan or Inf. Might be an easy one to tackle and prevent crashes due to NaN and Inf.

check_parameters_default_constructible might require a bit more work and restructuring. It is mostly about checking that given an instance you can construct a new one using only what is in get_params and only passing required arguments to the constructor. It also checks that only basic types are used for the arguments. I think sometimes we violate this, so it might take more effort and deprecations to start passing this check.

Overall I think we should tackle to checks related to checking and setting parameters. Refactoring the reflection system might already go a long way towards this. The tags related check is probably easy/quick to do. Checking for nan/inf also seems like it should be quick, but I somehow sense a trap :-/