Skip to content

Commit 712a2af

Browse files
Han Wangcharlesbluca
Han Wang
and
charlesbluca
authored
Set Dask-sql as the default Fugue Dask engine when installed (#640)
* Set Dask-sql as the default Fugue Dask engine when installed * Set Dask-sql as the default Fugue Dask engine when installed * Add max version constraint for `fugue` (#639) * Remove antlr4-python3-runtime constraint from 3.9+ test envs * Revert "Remove antlr4-python3-runtime constraint from 3.9+ test envs" This reverts commit ef30656. * Add max version constraint for fugue in 3.9+ envs * Constrain Fugue in remaining env/setup files * Clarify fugue constraint comments * Add pinning back to python 3.8 jdk11 tests * More reversions to python 3.8 jdk11 testing env * update * update * update * fix tests * update tests * update a few things * update * fix * conda install fugue in testing envs * Remove diff from features notebook * Alter documentation to mention automatic registration of execution engine * Expand FugueSQL notebook * Don't manually close client in simple statement test Co-authored-by: Charles Blackmon-Luca <[email protected]>
1 parent 20daf89 commit 712a2af

File tree

13 files changed

+623
-77
lines changed

13 files changed

+623
-77
lines changed

‎continuous_integration/environment-3.10-jdk11-dev.yaml‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ dependencies:
66
- dask-ml>=2022.1.22
77
- dask>=2022.3.0
88
- fastapi>=0.69.0
9+
- fugue>=0.7.0
910
- intake>=0.6.0
1011
- jpype1>=1.0.2
1112
- jsonschema
@@ -31,14 +32,3 @@ dependencies:
3132
- tpot
3233
- tzlocal>=2.1
3334
- uvicorn>=0.11.3
34-
# fugue dependencies; remove when we conda install fugue
35-
- adagio
36-
- antlr4-python3-runtime<4.10
37-
- ciso8601
38-
- fs
39-
- pip
40-
- qpd
41-
- triad
42-
- pip:
43-
# FIXME: tests are failing with fugue 0.7.0
44-
- fugue[sql]>=0.5.3,<0.7.0

‎continuous_integration/environment-3.10-jdk8-dev.yaml‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ dependencies:
66
- dask-ml>=2022.1.22
77
- dask>=2022.3.0
88
- fastapi>=0.69.0
9+
- fugue>=0.7.0
910
- intake>=0.6.0
1011
- jpype1>=1.0.2
1112
- jsonschema
@@ -31,14 +32,3 @@ dependencies:
3132
- tpot
3233
- tzlocal>=2.1
3334
- uvicorn>=0.11.3
34-
# fugue dependencies; remove when we conda install fugue
35-
- adagio
36-
- antlr4-python3-runtime<4.10
37-
- ciso8601
38-
- fs
39-
- pip
40-
- qpd
41-
- triad
42-
- pip:
43-
# FIXME: tests are failing with fugue 0.7.0
44-
- fugue[sql]>=0.5.3,<0.7.0

‎continuous_integration/environment-3.8-jdk11-dev.yaml‎

Lines changed: 1 addition & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ dependencies:
66
- dask-ml=2022.1.22
77
- dask=2022.3.0
88
- fastapi=0.69.0
9+
- fugue=0.7.0
910
- intake=0.6.0
1011
- jpype1=1.0.2
1112
- jsonschema
@@ -31,13 +32,3 @@ dependencies:
3132
- tpot
3233
- tzlocal=2.1
3334
- uvicorn=0.11.3
34-
# fugue dependencies; remove when we conda install fugue
35-
- adagio
36-
- antlr4-python3-runtime<4.10
37-
- ciso8601
38-
- fs
39-
- pip
40-
- qpd
41-
- triad
42-
- pip:
43-
- fugue[sql]==0.5.3

‎continuous_integration/environment-3.8-jdk8-dev.yaml‎

Lines changed: 1 addition & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ dependencies:
66
- dask-ml=2022.1.22
77
- dask=2022.3.0
88
- fastapi=0.69.0
9+
- fugue=0.7.0
910
- intake=0.6.0
1011
- jpype1=1.0.2
1112
- jsonschema
@@ -31,13 +32,3 @@ dependencies:
3132
- tpot
3233
- tzlocal=2.1
3334
- uvicorn=0.11.3
34-
# fugue dependencies; remove when we conda install fugue
35-
- adagio
36-
- antlr4-python3-runtime<4.10
37-
- ciso8601
38-
- fs
39-
- pip
40-
- qpd
41-
- triad
42-
- pip:
43-
- fugue[sql]==0.5.3

‎continuous_integration/environment-3.9-jdk11-dev.yaml‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ dependencies:
66
- dask-ml>=2022.1.22
77
- dask>=2022.3.0
88
- fastapi>=0.69.0
9+
- fugue>=0.7.0
910
- intake>=0.6.0
1011
- jpype1>=1.0.2
1112
- jsonschema
@@ -31,14 +32,3 @@ dependencies:
3132
- tpot
3233
- tzlocal>=2.1
3334
- uvicorn>=0.11.3
34-
# fugue dependencies; remove when we conda install fugue
35-
- adagio
36-
- antlr4-python3-runtime<4.10
37-
- ciso8601
38-
- fs
39-
- pip
40-
- qpd
41-
- triad
42-
- pip:
43-
# FIXME: tests are failing with fugue 0.7.0
44-
- fugue[sql]>=0.5.3,<0.7.0

‎continuous_integration/environment-3.9-jdk8-dev.yaml‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ dependencies:
66
- dask-ml>=2022.1.22
77
- dask>=2022.3.0
88
- fastapi>=0.69.0
9+
- fugue>=0.7.0
910
- intake>=0.6.0
1011
- jpype1>=1.0.2
1112
- jsonschema
@@ -31,14 +32,3 @@ dependencies:
3132
- tpot
3233
- tzlocal>=2.1
3334
- uvicorn>=0.11.3
34-
# fugue dependencies; remove when we conda install fugue
35-
- adagio
36-
- antlr4-python3-runtime<4.10
37-
- ciso8601
38-
- fs
39-
- pip
40-
- qpd
41-
- triad
42-
- pip:
43-
# FIXME: tests are failing with fugue 0.7.0
44-
- fugue[sql]>=0.5.3,<0.7.0

‎dask_sql/integrations/fugue.py‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
try:
22
import fugue
33
import fugue_dask
4+
from dask.distributed import Client
45
from fugue import WorkflowDataFrame, register_execution_engine
56
from fugue_sql import FugueSQLWorkflow
7+
from triad import run_at_def
68
from triad.utils.convert import get_caller_global_local_vars
79
except ImportError: # pragma: no cover
810
raise ImportError(
@@ -15,9 +17,25 @@
1517

1618
from dask_sql.context import Context
1719

18-
register_execution_engine(
19-
"dask", lambda conf: DaskSQLExecutionEngine(conf), on_dup="overwrite"
20-
)
20+
21+
@run_at_def
22+
def _register_engines() -> None:
23+
"""Register (overwrite) the default Dask execution engine of Fugue. This
24+
function is invoked as an entrypoint, users don't need to call it explicitly.
25+
"""
26+
register_execution_engine(
27+
"dask",
28+
lambda conf, **kwargs: DaskSQLExecutionEngine(conf=conf),
29+
on_dup="overwrite",
30+
)
31+
32+
register_execution_engine(
33+
Client,
34+
lambda engine, conf, **kwargs: DaskSQLExecutionEngine(
35+
dask_client=engine, conf=conf
36+
),
37+
on_dup="overwrite",
38+
)
2139

2240

2341
class DaskSQLEngine(fugue.execution.execution_engine.SQLEngine):

‎docs/environment.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,7 @@ dependencies:
1010
- maven>=3.6.0
1111
- dask>=2022.3.0
1212
- pandas>=1.1.2
13-
# FIXME: tests are failing with fugue 0.7.0
14-
- fugue>=0.5.3,<0.7.0
13+
- fugue>=0.7.0
1514
- jpype1>=1.0.2
1615
- fastapi>=0.69.0
1716
- uvicorn>=0.11.3

‎docs/requirements-docs.txt‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,7 @@ sphinx-tabs
33
dask-sphinx-theme>=3.0.0
44
dask>=2022.3.0
55
pandas>=1.1.2
6-
# FIXME: tests are failing with fugue 0.7.0
7-
fugue>=0.5.3,<0.7.0
6+
fugue>=0.7.0
87
jpype1>=1.0.2
98
fastapi>=0.69.0
109
uvicorn>=0.11.3

‎docs/source/fugue.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ In order to offer a "best of both worlds" solution, dask-sql includes several op
88
dask-sql as a FugueSQL engine
99
-----------------------------
1010

11-
FugueSQL users unfamiliar with dask-sql can take advantage of its functionality with minimal code changes by passing :class:`dask_sql.integrations.fugue.DaskSQLExecutionEngine` into the ``FugueSQLWorkflow`` being used to execute commands.
11+
FugueSQL users unfamiliar with dask-sql can take advantage of its functionality by installing it in an environment alongside Fugue; this will automatically register :class:`dask_sql.integrations.fugue.DaskSQLExecutionEngine` as the default Dask execution engine for FugueSQL queries.
1212
For more information and sample usage, see `Fugue — dask-sql as a FugueSQL engine <https://fugue-tutorials.readthedocs.io/tutorials/integrations/dasksql.html>`_.
1313

1414
Using FugueSQL on an existing ``Context``

0 commit comments

Comments
 (0)