[Tracker] Test Infrastructure Improvements

[csadorf]

Context and Motivation

The cuML test infrastructure has evolved organically as the library has grown, leading to several challenges including test flakiness, coverage gaps, and inconsistent fixture usage. A systematic overhaul of the test infrastructure will help address these issues by improving reliability, clarifying test organization, optimizing execution time, and reducing maintenance burden - ultimately providing a better development experience and more robust codebase.

It is assumed that the plan outlined here will have to be implemented incrementally. The issue serves to tracking overarching aims and progress.

Overall Aims

1. Reduce test flakiness
2. Increase test coverage
3. Improve test maintainability and organization
4. Enhance test performance and execution time

General Approach

1. Better delineate between functional tests that check the plumping and correctness tests that check whether our algorithmic implementation is correct.
2. For correctness checks threshold should ideally be rooted in an analytical expectation based on the condition of the problem, but realistically, be more aggressive about reducing thresholds to levels that would clearly indicate a regression rather than barely below the usually observed value.
3. Make sure that especially the correctness checks are as deterministic as possible while keeping in mind that in many cases it simply isn’t due to the nature of parallel computation on GPU devices.
4. Use per-test retries where appropriate, i.e., for correctness tests where single test failure among many successes does not immediately indicate a regression.
5. Use automated retries for all components that rely on external resources, e.g., dataset downloads.

Specific Approach

1. Test Organization and Structure

Test Categories

• Consider all tests as functional by default (API, input validation, basic behavior)
• Explicitly mark or identify:
  • Correctness tests (algorithmic implementation verification)
  • Integration tests (end-to-end workflows)
• Keep performance tests separate in dedicated benchmarking suite

Organization Alternatives

1. Test Type Identification Options
   a. Via pytest markers:

   @pytest.mark.correctness
   @pytest.mark.integration

   • Pros: Flexible filtering, no file reorganization needed
   • Cons: Requires discipline in marker usage

   b. Via filename conventions:

   test_kmeans.py           # functional tests (default)
   test_kmeans_correct.py   # correctness tests
   test_kmeans_integ.py     # integration tests
   

   • Pros: Clear visual separation
   • Cons: Could lead to code duplication

   c. Via test name conventions:

   def test_kmeans_fit()          # functional test
   def test_correct_kmeans_conv() # correctness test
   def test_integ_kmeans_pipe()   # integration test

   • Pros: Easy to implement
   • Cons: Less structured than other options

2. Directory-Based Structure

   tests/
     functional/    # default location
     correctness/
     integration/
   

   • Pros: Clear separation of concerns
   • Cons: May require significant reorganization

Recommendations for Review

1. Evaluate current test organization pain points
2. Consider implementing pytest markers as a non-invasive first step
3. Review potential benefits of filename conventions
4. Assess need for directory restructuring based on maintenance experience

2. Test Quality Improvements

• For functional tests:

  • Ensure comprehensive input validation (use hypothesis strategies where possible)
  • Test edge cases and error conditions
  • Verify API contract compliance
  • Avoid thresholded checks or use very conservative thresholds

• For correctness checks:

  • Root thresholds in analytical expectations where possible
  • Set aggressive thresholds that clearly indicate regressions
  • Document the rationale behind threshold choices and expected variance
  • Try to implement tests as deterministic as possible
  • Use automatic retries where appropriate*

*) In case that we opt to use a "correctness" marker, retry logic could be implemented as part of the marker.

4. Test Infrastructure Enhancements

• More consistent use of common fixtures (deduplicate existing ones)
• More consistent use of hypothesis strategies for input validation

Related issues

• Use pytest-randomly to run tests in a random order #6375
• Cleanup warnings in test suite #6366
• [BUG] conda-python-tests-singlegpu taking 4+ hours to complete #6251
• [TRACKER] Flaky tests tracker #3744
• [FEA] Update threshold used in RandomForest tests #2707
• [FEA] Create tests for soft vs hard dependencies #5917
• [FEA] Reduce overparameterization of tests in test_datasets.py #3081
• [FEA] Expand hypothesis testing to all suitable tests for linear models. #4964

[csadorf]

Additional points to consider:

1. cuML accuracy tests should more often than not compare directly against hard-coded (recorded) values. Those can originate from a reference implementation, e.g., scikit-learn, but do not have to. In either case their origin must be clearly stated, e.g., via a comment.
2. cuML.accel accuracy tests should ideally be compared directly to the accelerated/proxied reference implementation (potentially with low precision to account for numerical differences), but do not have to in cases where there are significant differences in the underlying algorithm, but results are nevertheless equivalent.
3. In cases where results are only equivalently similarly, i.e., they differ significantly in value, we must test the quality of results through an appropriate metric (e.g., trustworthiness for manifold preservation, etc.).

Point 1 is motivated by our aim to make scikit-learn a hard dependency with a wide version compatibility range (#6437 ).

[jcrist]

I'm -1 on anything that splits tests for a single estimator between multiple files. There are a few exceptions:

• Performance testing/benchmarking, as those require different infrastructure and make more sense to isolate in a different directory.
• Generic estimator checks like we have in test_base.py currently (pickleable, get_params/set_params/...). These tests check behavior common across all estimators, and don't need to branch on any estimator-specifics.

It's not easy to distinguish between functional/correctness/integration testing - I worry that fracturing our tests further into multiple files will:

• Make it harder to know where to put a single test
• Likewise harder to find the test later if we're looking for something that checks an existing behavior
• Harder to visually inspect if a bit of an estimator is already tested (e.g. "what parameterization coverage do we have for build_algo="nn_descent" for umap?")
• Lead to accidental test duplication between files
• Make it harder to run tests for a single estimator since they're no longer isolated to a single file.

Further, I'm not sure I see the benefit of labeling the difference between "functional" and "correctness" tests? It's good hygiene to write smaller tests that check the behavior you're looking for (rather than one massive series of calls and asserts), but often tests will effectively be checking both plumbing and correctness in some form. Good test hygiene is important, but like the above I worry that labels will add visual noise without additional benefit (and force developers to be grouping tests into groups that may feel hard to distinguish or separate).

Can you give an example of what you mean by "integration test" in the context of cuml outside of cuml.accel or performance testing? I can't think of what this would mean in our codebase.

*) In case that we opt to use a "correctness" marker, retry logic could be implemented as part of the marker.

Unless the operation we're testing can fail for known reasons (network, etc...), I'm very against papering over test flakiness in correctness with retries. We should strive to write our tests to be reproducible and have checks with sufficient accuracy ranges that this isn't necessary.

---

I think what I'd like to see is:

• Removal of large scale runs from any of our standard tests (e.g. stress_param). Ideally scale wouldn't even need to be parametrized in these tests unless it triggers alternate code paths - we should try to check correctness with small datasets to ensure fast runs. This matches above with splitting out performance testing from our standard tests.
• Recommendations for best tool to use for which case. When does hypothesis make sense? When should we use pytest.parametrize instead?
• Better test hygiene - prefer smaller more specific tests over large hypothesis-ized workflows which are less clear what they're checking.
• Split out common tests into either reusable functions like sklearn's check_estimator, or standard tests like we have in test_base. I have a slight preference for the latter provided we don't need to make exceptions per-estimator. The former makes more sense if the checks are likely slightly different per estimator.
• Better correctness testing (I think the recommendations you have above are good!)

I suspect a good way to do this is to pick a single test file and clean it up, noting what changes were needed as you do so. Once we can agree on what "good" looks like, we can write down some recommendations, then parallelize further work over time.

[csadorf]

We should consider more rigorously tracking memory usage in our CI runners.

[csadorf]

Relevant issue on runtime analysis of our tests: #5053