Drop support for Python 3.9

[bdice]

This issue tracks dropping Python 3.9, following from the discussion on #40:

When should we drop Python 3.9?

Typically RAPIDS has kept the matrix of supported Python minor versions to 2 or 3 versions at a time. When adding Python 3.12, we should probably drop Python 3.9 as well.

SPEC 0 recommended dropping support for Python 3.9 in 2023Q4. Meanwhile, NEP 29 recommended dropping support for Python 3.9 as of Apr 05, 2024. Both of these deadlines have passed and several large Python libraries are now moving towards dropping Python 3.9, so it is probably reasonable to drop Python 3.9 around the same time that we add Python 3.12.

We are beyond the SPEC 0 date for dropping Python 3.9 now, and expect that RAPIDS dependencies will gradually drop support. We are exploring dropping Python 3.9 in RAPIDS 24.10.

Tasks:

• Announce deprecation: Dropping Support for Python v3.9 and adding Python 3.12 Support in Release v24.10 docs#528
• Drop Python 3.9 from shared-workflows: Drop Python 3.9 from workflows. shared-workflows#235
• For each repo, update Python requirements
  • Update matrix filters referencing Python 3.9
  • Edit dependencies.yaml to remove py: 3.9 sections like this
  • Edit dependencies.yaml to raise the fallback matrix's Python lower bound
  • Update CMake references like find_package(Python 3.9 ...)
  • Update pyproject.toml: requires-python and specifiers
  • Check for special cases, like ucxx's Python build matrix in conda_build_config.yaml
  • Update README, CONTRIBUTING, and docs as needed
  • grep for 3.9 to make sure nothing was missed!

Things to be careful about:

• Python version embedded in Jupyter notebook metadata (like this)
• avoid changing benchmarking results and the code to reproduce them
• pynvjitlink is used by more than just RAPIDS (like https://github.com/NVIDIA/nvmath-python), and uses its own CI matrix to support a broader range of versions than RAPIDS (link)... we may sometimes drop a Python version across RAPIDS but still support it in pynvjitlink
• generic tools like rapids-reviser, rapids-build-backend, rapids-metadata, etc. shouldn't have their floors bumped unless there's a compelling reason to do so... that'd make it harder to fix bugs in those tools that affect older releases

### before making changes in individual repos
- [x] Announce deprecation: https://github.com/rapidsai/docs/pull/528
- [x] Drop Python 3.9 from shared-workflows: https://github.com/rapidsai/shared-workflows/pull/235

### Updates for libraries
- [x] cucim (https://github.com/rapidsai/cucim/pull/766)
- [x] cudf (https://github.com/rapidsai/cudf/pull/16637)
- [x] cugraph (https://github.com/rapidsai/cugraph/pull/4625)
- [x] cugraph-gnn (https://github.com/rapidsai/cugraph-gnn/pull/32)
- [x] cugraph-pg (https://github.com/rapidsai/cugraph-pg/pull/25 ... this repo is not being actively developed right now, not waiting for that to be mered to consider this done)
- [x] cuml (https://github.com/rapidsai/cuml/pull/6040)
- [x] cuspatial (https://github.com/rapidsai/cuspatial/pull/1443)
- [x] cuvs (https://github.com/rapidsai/cuvs/pull/301)
- [x] cuxfilter (https://github.com/rapidsai/cuxfilter/pull/623)
- [x] dask-cuda (https://github.com/rapidsai/dask-cuda/pull/1377)
- [x] kvikio (https://github.com/rapidsai/kvikio/pull/443)
- [x] pynvjitlink (https://github.com/rapidsai/pynvjitlink/pull/102)
- [x] rapids-cmake (https://github.com/rapidsai/rapids-cmake/pull/684)
- [x] raft (https://github.com/rapidsai/raft/pull/2417)
- [x] rmm (https://github.com/rapidsai/rmm/pull/1659)
- [x] ucx-py (https://github.com/rapidsai/ucx-py/pull/1065)
- [x] ucxx (https://github.com/rapidsai/ucxx/pull/272)
- [x] wholegraph (https://github.com/rapidsai/wholegraph/pull/209)
- [x] private repos

### after all libraries are updated
- [x] `pypi-wheel-scripts` (https://github.com/rapidsai/pypi-wheel-scripts/pull/16)
- [x] `ci-imgs` (https://github.com/rapidsai/ci-imgs/pull/173)
- [x] dask-build-environment (https://github.com/rapidsai/dask-build-environment/pull/97)
- [x] update install docs? (https://github.com/rapidsai/docs/blob/01dbbcb67c199893b839651f5dca01356c825068/install/install.md?plain=1#L101) (https://github.com/rapidsai/docs/pull/534)
- [x] Update release selector: need to disable Python 3.9 on nightlies https://github.com/rapidsai/docs/blob/01dbbcb67c199893b839651f5dca01356c825068/_includes/selector.html (https://github.com/rapidsai/docs/pull/534)

[jakirkham]

Some quick data points:

• NumPy 2.1.0 drops Python 3.9 (note 2.0.0 has 3.9 support)
  • ENH: Drop support for Python 3.9? numpy/numpy#26242
• Pandas next release drops Python 3.9 (cites SPEC 0)
  • DEPS: Drop Python 3.9 pandas-dev/pandas#58238
• CuPy 14 plans to drop Python 3.9 (cites SPEC 0)
  • CuPy v14 Release Plan cupy/cupy#8269
• Numba still supports Python 3.9
• scikit-learn still supports Python 3.9
• scikit-image still supports Python 3.9
• NetworkX 3.3 dropped Python 3.9 (came out Oct 2023)
  • Drop Python 3.9 support networkx/networkx#7028
• Bokeh 3.5 dropped Python 3.9
  • Drop support for Python 3.9 and modernize the codebase bokeh/bokeh#13634
• Dask is discussing dropping Python 3.9
  • End of Python3.9 support dask/community#373
• XGBoost next release drops Python 3.9
  • [breaking] Bump Python requirement to 3.10. dmlc/xgboost#10434

[jameslamb]

I'm +1 on doing this in this timeframe.

DLFW builds are using Python 3.10 (DLFW support matrix), as that's the stable python3 that ships with Ubuntu 22.04 (Python version across Linux distros), so this change shouldn't be disruptive for those builds.

And nemo-curator is on python >= 3.10 (code link), so they shouldn't be affected either.

[rjzamora]

We are exploring dropping Python 3.9 in RAPIDS 24.10.

Just curious to know what the ETA is for dropping Python 3.9?

I believe we will need to remain pinned to dask 2024.7.1 until then. In the meantime, we cannot catch breaking changes in dask/distributed.

[jameslamb]

The plan is to do this "within 24.10", but we can't be more precise than that.

I believe we will need to remain pinned to dask 2024.7.1 until then. In the meantime, we cannot catch breaking changes in dask/distributed.

For Python 3.9, yes, but I think rapids-dask-dependency could take on a dependency on newer dask / distributed for Python 3.10+

Like this:

# TODO: drop the python_version specifiers and just use latest versions built from GitHub once
#              RAPIDS drops Python 3.9 support
dependencies = [
    "dask @ git+https://github.com/dask/dask.git@main ; python_version >= '3.10'",
    "distributed @ git+https://github.com/dask/distributed.git@main ; python_version >= '3.10'",
    "dask-expr @ git+https://github.com/dask/dask-expr.git@main ; python_version >= '3.10'",
    "dask==2024.7.1 ; python_version < '3.10'",
    "distributed==2024.7.1 ; python_version < '3.10'",
    "dask-expr ; python_version < '3.10'",
]

That'd allow projects to start testing against the latest development version of Dask throughout the 24.10 release cycle.

[rjzamora]

That'd allow projects to start testing against the latest development version of Dask throughout the 24.10 release cycle.

Okay, thanks for the response @jameslamb! This may be our best option -- I'll look into it.

[jakirkham]

For rapids-dask-dependency, likely Conda will work ok as-is since the solver will pick packages that are Python 3.9 (or 3.10+) compatible. Though please let us know if not

[bdice]

Dropping 3.9 builds should now be safe, and would also unblock the work of updating our dask pinning.

This PR does that: rapidsai/shared-workflows#235

This change in shared-workflows is also safe to merge before we do the cleanup in each repo to remove dependencies, configs, etc. referencing Python 3.9. We're just building/testing fewer packages.