[WIP] Make use of load_setup_py_data to have less/no duplication of packages

[galipremsagar]

This PR is WIP.

[jakirkham]

Thanks Prem! 🙏

We did this for Dask-CUDA in PR ( rapidsai/dask-cuda#893 ) if that helps 🙂

[bdice]

🤔 Pinning this isn't very friendly to source installs, since it relies on CI environment variables. For example, I don't think you could install a cudf package from conda and install dask-cudf from source with this construction. Do we want to couple these packages' installation methods that closely?

[github-actions]

This PR has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this PR if it is no longer required. Otherwise, please respond with a comment indicating any updates. This PR will be labeled inactive-90d if there is no activity in the next 60 days.

[github-actions]

This PR has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this PR if it is no longer required. Otherwise, please respond with a comment indicating any updates.

[jakirkham]

FWIW this will probably need to change when we move to pyproject.toml. We did this recently in Dask-CUDA and it wasn't too difficult. Would take a look at this commit ( rapidsai/dask-cuda@72bbfa4 ) or PR ( rapidsai/dask-cuda#1035 ) generally as a guide

[vyasr]

@galipremsagar we should discuss how best to proceed here, and in particular how best to synchronize this with the work on the dependency file generator. I see that there is some logic in this PR to filter out cupy-cuda from the setup.py dependency list. In the interest of having a single source of truth (the new dependencies.yaml file), I think we would be better served by declaring the right groups of dependencies for each file type determined entirely by dependencies.yaml (including any differences between e.g. conda's cupy and PyPI's cupy-cuda11x). We may still want to leverage functions like load_file_data, but not load_setup_py_data since the management of dependency differences between the two would be handled upstream. @bdice I see that you took a look at this PR as well already, what do you think?

[vyasr]

A lot has changed since this PR was originally opened. We now use rapids-dependency-file-generator for most dependency generation (meta.yaml remains outstanding), and all Python package metadata has been moved to pyproject.toml. My guess is that at this point even if we wanted to move forward with this type of metadata sharing it would require a new implementation, so I'm going to close this PR.