Update nightly recipe / setup for `2021.11.0` release #308

charlesbluca · 2021-11-11T14:39:50Z

As part of the new release, we made some modifications to the conda recipe using grayskull, so that it now generally matches up with the setup.py. This PR reflects some of those changes on the nightly recipe, and also makes some updates to the setup.py so that it is consistent with the desired conda recipe.

charlesbluca · 2021-11-11T14:41:04Z

continuous_integration/recipe/meta.yaml

+  entry_points:
+    - dask-sql-server = dask_sql.server.app:main
+    - dask-sql = dask_sql.cmd:main
+  noarch: python


Should we still build nightlies noarch even though the stable releases don't do this?

No this should be dropped. It doesn't work with the selector below

Just for clarity, does this mean that this package will only be available to the architecture + Python version used to build here (linux-64, py3.8)? If so then we might want to open a larger issue on who is using the nightlies to assess if this is desired or if we have to rethink our workflow.

If not, then we should be good to just remove this line and proceed

Will need to use variants to construct a build matrix for the Python versions

No easy answers for OS other than including more jobs on other OSes

Am assuming we are the primary users of nightlies. So would use that as a guiding principle until someone else asks for something more

Thanks for the link - I'll play around with variants for now, and check in with RAPIDS folk to see if we have any specific needs here.

charlesbluca · 2021-11-11T14:41:51Z

setup.py

 setup(
    name="dask_sql",
-    description="Dask SQL",
+    description="SQL query layer for Dask",


Just noting that this change maps to the conda recipe's summary.

charlesbluca · 2021-11-11T18:45:47Z

setup.py

    setup_requires=["setuptools_scm"] + sphinx_requirements,
    install_requires=[
-        "dask[dataframe,distributed]==2021.10.0",
+        "dask[dataframe,distributed]>=2021.10.0",


We'll need this if we want to use unpinned versions of Dask with the nightlies for certain features (dask-sql-server in particular)

codecov-commenter · 2021-11-11T18:54:45Z

Codecov Report

Merging #308 (ead3d94) into main (2194a75) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #308   +/-   ##
=======================================
  Coverage   95.75%   95.75%           
=======================================
  Files          65       65           
  Lines        2802     2802           
  Branches      418      418           
=======================================
  Hits         2683     2683           
  Misses         77       77           
  Partials       42       42

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2194a75...ead3d94. Read the comment docs.

jakirkham

Given the recipe changes, noarch needs to be dropped

jakirkham · 2021-11-11T19:00:30Z

continuous_integration/recipe/meta.yaml

+  entry_points:
+    - dask-sql-server = dask_sql.server.app:main
+    - dask-sql = dask_sql.cmd:main
+  noarch: python


No this should be dropped. It doesn't work with the selector below

jakirkham · 2021-11-11T19:00:33Z

continuous_integration/recipe/meta.yaml

+    - pygments
+    - nest-asyncio
+    - tabulate
+    - importlib-metadata  # [py<38]


This is the selector that makes noarch not an option

jakirkham

This should just work if the variant name shares the package name

continuous_integration/recipe/conda_build_config.yaml

continuous_integration/recipe/meta.yaml

Co-authored-by: jakirkham <[email protected]>

charlesbluca

Missed something

charlesbluca · 2021-11-11T19:27:04Z

continuous_integration/recipe/meta.yaml

+  entry_points:
+    - dask-sql-server = dask_sql.server.app:main
+    - dask-sql = dask_sql.cmd:main
+  string: py{{ py_version.replace(".", "") }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }}


Forgot about this - is there a clean way to get a Python version for the package string from the build variant?

Can just replace with python instead of py_version. Though maybe you have already done this :)

charlesbluca · 2021-11-11T19:27:20Z

continuous_integration/recipe/meta.yaml

@@ -1,6 +1,5 @@
 {% set name = "dask-sql" %}
 {% set version = environ.get('GIT_DESCRIBE_TAG', '0.0.0.dev') + environ.get('VERSION_SUFFIX', '') %}
-{% set py_version=environ.get('CONDA_PY', 36) %}


If not will probably need to add this back in and modify

continuous_integration/recipe/meta.yaml

Co-authored-by: jakirkham <[email protected]>

charlesbluca · 2021-11-11T19:48:58Z

Maybe we need to do the replace in a set template at the top of the file?

jakirkham · 2021-11-11T19:54:56Z

Shouldn't be needed

charlesbluca · 2021-11-11T19:55:55Z

Think the issue might be that we need to use a replace filter here? If these are jinja2 templates that is

jakirkham · 2021-11-11T19:57:00Z

Yeah saw the latest commit. That seems like what is needed

continuous_integration/recipe/meta.yaml

jakirkham

Couple minor suggestions

setup.py

charlesbluca · 2021-11-11T20:15:44Z

Looks like the conda build job takes roughly twice as long, which isn't as bad as I was expecting. I think this should be good to merge in pending tests, and we can follow up with a different option if other OSes are desired (could we do MacOS / Windows builds on gpuCI?)

If timing ever becomes an issue, I wonder if it's possible / safe to only build the package for one Python version on PRs

jakirkham · 2021-11-11T20:52:57Z

Thanks Charles! 😄

jakirkham · 2021-11-11T21:00:25Z

Yeah to your point above about build time. We could separate out the matrix onto multiple CI jobs. This is effectively what conda-forge does. It contains a bunch of variants and builds each one on a separate CI job. It's certainly something we could pursue if it becomes an issue.

Big picture, think it really all depends how long we maintain the Python 3.7 build here. Guessing we will want to add 3.9 & 3.10, which would increase the build matrix and may motivating dropping older Python versions. Dask (and others) may also choose to drop 3.7 when adding 3.10, which would push us in the same direction. Once 3.7 is dropped we don't really need the importlib-metadata dependency and can return to noarch (assuming we don't add any other Python version specific dependencies). At which point we don't really need to worry about more sophisticated builds.

charlesbluca added 2 commits November 11, 2021 06:25

Update conda recipe to match grayskull output

9d2d302

Update package description in setup.py

8d05148

charlesbluca commented Nov 11, 2021

View reviewed changes

Unpin dask dependency for development

81fd84a

charlesbluca commented Nov 11, 2021

View reviewed changes

jakirkham reviewed Nov 11, 2021

View reviewed changes

charlesbluca added 2 commits November 11, 2021 11:21

Try out python version variants

f5851ba

Remove noarch line

7e139e0

jakirkham reviewed Nov 11, 2021

View reviewed changes

continuous_integration/recipe/conda_build_config.yaml Outdated Show resolved Hide resolved

continuous_integration/recipe/meta.yaml Outdated Show resolved Hide resolved

Apply suggestions from code review

6d4593d

Co-authored-by: jakirkham <[email protected]>

charlesbluca commented Nov 11, 2021

View reviewed changes

charlesbluca added 2 commits November 11, 2021 11:37

Bring back manually computed py_version

6ae9d4a

Use python variant for package string name

38f03ce

jakirkham reviewed Nov 11, 2021

View reviewed changes

continuous_integration/recipe/meta.yaml Outdated Show resolved Hide resolved

Remove dot from python string version

903cbe6

Co-authored-by: jakirkham <[email protected]>

charlesbluca added 2 commits November 11, 2021 11:51

Set python version string at top of file

3f239fc

Use jinja2 replace filter

60da6ca

jakirkham reviewed Nov 11, 2021

View reviewed changes

continuous_integration/recipe/meta.yaml Outdated Show resolved Hide resolved

jakirkham reviewed Nov 11, 2021

View reviewed changes

setup.py Outdated Show resolved Hide resolved

Use HTTPS for project URL

a674972

charlesbluca added 2 commits November 11, 2021 12:20

Upload linux packages instead of noarch

5c4d069

Add maintainers to recipe

ead3d94

jakirkham approved these changes Nov 11, 2021

View reviewed changes

jakirkham merged commit ae499a5 into dask-contrib:main Nov 11, 2021

charlesbluca deleted the update-nightly-recipe branch January 19, 2022 21:23

Update nightly recipe / setup for 2021.11.0 release #308

Update nightly recipe / setup for 2021.11.0 release #308

Uh oh!

Conversation

charlesbluca commented Nov 11, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Nov 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakirkham left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

charlesbluca left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charlesbluca commented Nov 11, 2021

Uh oh!

jakirkham commented Nov 11, 2021

Uh oh!

charlesbluca commented Nov 11, 2021

Uh oh!

jakirkham commented Nov 11, 2021

Uh oh!

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charlesbluca commented Nov 11, 2021

Uh oh!

jakirkham commented Nov 11, 2021

Uh oh!

jakirkham commented Nov 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Update nightly recipe / setup for `2021.11.0` release #308

Update nightly recipe / setup for `2021.11.0` release #308

codecov-commenter commented Nov 11, 2021 •

edited

Loading

jakirkham left a comment •

edited

Loading