Skip to content

[REVIEW] Fix unary conditional join operations #366

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
charlesbluca merged 8 commits into from
Jan 24, 2022
Merged

[REVIEW] Fix unary conditional join operations #366

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c9f4534
add test case that causes processing to hang indefinately
jdye64 Jan 11, 2022
d5a05ba
Added logic to check for unary join conditions like IS NOT NULL
jdye64 Jan 11, 2022
e1fd2b8
Remove PROJECT_REDUCE_EXPRESSION since RelOpts with no operands cause…
jdye64 Jan 13, 2022
089b538
updates per review
jdye64 Jan 13, 2022
6813106
remove accidental driver Java class that was added for testing
jdye64 Jan 13, 2022
2aed875
address review comments
jdye64 Jan 14, 2022
8f2b386
Removed JoinPushThroughRules as certain queries joins would be optimi…
jdye64 Jan 19, 2022
5843594
Merge remote-tracking branch 'upstream/main' into stackoverflow_optim…
charlesbluca Jan 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions dask_sql/physical/rel/logical/join.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,15 +259,16 @@ def _split_join_condition(
filter_condition = []

for operand in operands:
try:
lhs_on_part, rhs_on_part = self._extract_lhs_rhs(operand)
lhs_on.append(lhs_on_part)
rhs_on.append(rhs_on_part)
continue
except AssertionError:
pass

filter_condition.append(operand)
if isinstance(operand, org.apache.calcite.rex.RexCall):
try:
lhs_on_part, rhs_on_part = self._extract_lhs_rhs(operand)
lhs_on.append(lhs_on_part)
rhs_on.append(rhs_on_part)
continue
except AssertionError:
pass

filter_condition.append(operand)

if lhs_on and rhs_on:
return lhs_on, rhs_on, filter_condition
Expand Down
20 changes: 20 additions & 0 deletions tests/integration/test_join.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,3 +183,23 @@ def test_join_literal(c):
df_expected = pd.DataFrame({"user_id": [], "b": [], "user_id0": [], "c": []})

assert_frame_equal(df.reset_index(), df_expected.reset_index(), check_dtype=False)


def test_conditional_join(c):
df1 = pd.DataFrame({"a": [1, 2, 2], "b": ["x", "y", "z"]})
df2 = pd.DataFrame({"c": [2, 3, 5], "d": ["i", "j", "k"]})

c.create_table("df1", df1)
c.create_table("df2", df2)

query = """
SELECT
a
FROM df1
INNER JOIN df2 ON
(
a = c
AND b IS NOT NULL
)
"""
ddf = c.sql(query)