-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base #25161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
[SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base #25161
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
80f2894
for SPARK-28390
shivusondur 624fad5
Handled th review comments
shivusondur b0a4fd1
Handled the few more comments
shivusondur fc65bcf
Handled the few more comments
shivusondur a1ba9a8
Just added the TODO comment
shivusondur ef39cdd
Removed the extra queries(made like original) and added udf() whereve…
shivusondur c4ad657
Added different combination of udf
shivusondur 8795d66
Only "TODO" comment is moved up
shivusondur File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
57 changes: 57 additions & 0 deletions
57
sql/core/src/test/resources/sql-tests/inputs/udf/pgSQL/udf-select_having.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| -- | ||
| -- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group | ||
| -- | ||
| -- | ||
| -- SELECT_HAVING | ||
| -- https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/select_having.sql | ||
| -- | ||
| -- This test file was converted from inputs/pgSQL/select_having.sql | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we add this todo on the top: and remove same lines below?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
| -- TODO: We should add UDFs in GROUP BY clause when [SPARK-28445] is resolved. | ||
|
|
||
HyukjinKwon marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| -- load test data | ||
| CREATE TABLE test_having (a int, b int, c string, d string) USING parquet; | ||
| INSERT INTO test_having VALUES (0, 1, 'XXXX', 'A'); | ||
| INSERT INTO test_having VALUES (1, 2, 'AAAA', 'b'); | ||
| INSERT INTO test_having VALUES (2, 2, 'AAAA', 'c'); | ||
| INSERT INTO test_having VALUES (3, 3, 'BBBB', 'D'); | ||
| INSERT INTO test_having VALUES (4, 3, 'BBBB', 'e'); | ||
| INSERT INTO test_having VALUES (5, 3, 'bbbb', 'F'); | ||
| INSERT INTO test_having VALUES (6, 4, 'cccc', 'g'); | ||
| INSERT INTO test_having VALUES (7, 4, 'cccc', 'h'); | ||
| INSERT INTO test_having VALUES (8, 4, 'CCCC', 'I'); | ||
| INSERT INTO test_having VALUES (9, 4, 'CCCC', 'j'); | ||
|
|
||
| SELECT udf(b), udf(c) FROM test_having | ||
| GROUP BY b, c HAVING udf(count(*)) = 1 ORDER BY udf(b), udf(c); | ||
|
|
||
| -- HAVING is effectively equivalent to WHERE in this case | ||
| SELECT udf(b), udf(c) FROM test_having | ||
| GROUP BY b, c HAVING udf(b) = 3 ORDER BY udf(b), udf(c); | ||
|
|
||
| -- [SPARK-28386] Cannot resolve ORDER BY columns with GROUP BY and HAVING | ||
| -- SELECT lower(c), count(c) FROM test_having | ||
| -- GROUP BY lower(c) HAVING count(*) > 2 OR min(a) = max(a) | ||
| -- ORDER BY lower(c); | ||
|
|
||
| SELECT udf(c), max(udf(a)) FROM test_having | ||
| GROUP BY c HAVING udf(count(*)) > 2 OR udf(min(a)) = udf(max(a)) | ||
| ORDER BY c; | ||
|
|
||
| -- test degenerate cases involving HAVING without GROUP BY | ||
| -- Per SQL spec, these should generate 0 or 1 row, even without aggregates | ||
|
|
||
| SELECT udf(udf(min(udf(a)))), udf(udf(max(udf(a)))) FROM test_having HAVING udf(udf(min(udf(a)))) = udf(udf(max(udf(a)))); | ||
| SELECT udf(min(udf(a))), udf(udf(max(a))) FROM test_having HAVING udf(min(a)) < udf(max(udf(a))); | ||
|
|
||
| -- errors: ungrouped column references | ||
| SELECT udf(a) FROM test_having HAVING udf(min(a)) < udf(max(a)); | ||
| SELECT 1 AS one FROM test_having HAVING udf(a) > 1; | ||
|
|
||
| -- the really degenerate case: need not scan table at all | ||
| SELECT 1 AS one FROM test_having HAVING udf(udf(1) > udf(2)); | ||
| SELECT 1 AS one FROM test_having HAVING udf(udf(1) < udf(2)); | ||
|
|
||
| -- and just to prove that we aren't scanning the table: | ||
| SELECT 1 AS one FROM test_having WHERE 1/udf(a) = 1 HAVING 1 < 2; | ||
|
|
||
| DROP TABLE test_having; | ||
187 changes: 187 additions & 0 deletions
187
sql/core/src/test/resources/sql-tests/results/udf/pgSQL/udf-select_having.sql.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,187 @@ | ||
| -- Automatically generated by SQLQueryTestSuite | ||
| -- Number of queries: 22 | ||
|
|
||
|
|
||
| -- !query 0 | ||
| CREATE TABLE test_having (a int, b int, c string, d string) USING parquet | ||
| -- !query 0 schema | ||
| struct<> | ||
| -- !query 0 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 1 | ||
| INSERT INTO test_having VALUES (0, 1, 'XXXX', 'A') | ||
| -- !query 1 schema | ||
| struct<> | ||
| -- !query 1 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 2 | ||
| INSERT INTO test_having VALUES (1, 2, 'AAAA', 'b') | ||
| -- !query 2 schema | ||
| struct<> | ||
| -- !query 2 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 3 | ||
| INSERT INTO test_having VALUES (2, 2, 'AAAA', 'c') | ||
| -- !query 3 schema | ||
| struct<> | ||
| -- !query 3 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 4 | ||
| INSERT INTO test_having VALUES (3, 3, 'BBBB', 'D') | ||
| -- !query 4 schema | ||
| struct<> | ||
| -- !query 4 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 5 | ||
| INSERT INTO test_having VALUES (4, 3, 'BBBB', 'e') | ||
| -- !query 5 schema | ||
| struct<> | ||
| -- !query 5 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 6 | ||
| INSERT INTO test_having VALUES (5, 3, 'bbbb', 'F') | ||
| -- !query 6 schema | ||
| struct<> | ||
| -- !query 6 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 7 | ||
| INSERT INTO test_having VALUES (6, 4, 'cccc', 'g') | ||
| -- !query 7 schema | ||
| struct<> | ||
| -- !query 7 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 8 | ||
| INSERT INTO test_having VALUES (7, 4, 'cccc', 'h') | ||
| -- !query 8 schema | ||
| struct<> | ||
| -- !query 8 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 9 | ||
| INSERT INTO test_having VALUES (8, 4, 'CCCC', 'I') | ||
| -- !query 9 schema | ||
| struct<> | ||
| -- !query 9 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 10 | ||
| INSERT INTO test_having VALUES (9, 4, 'CCCC', 'j') | ||
| -- !query 10 schema | ||
| struct<> | ||
| -- !query 10 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 11 | ||
| SELECT udf(b), udf(c) FROM test_having | ||
| GROUP BY b, c HAVING udf(count(*)) = 1 ORDER BY udf(b), udf(c) | ||
| -- !query 11 schema | ||
| struct<CAST(udf(cast(b as string)) AS INT):int,CAST(udf(cast(c as string)) AS STRING):string> | ||
| -- !query 11 output | ||
| 1 XXXX | ||
| 3 bbbb | ||
|
|
||
|
|
||
| -- !query 12 | ||
| SELECT udf(b), udf(c) FROM test_having | ||
| GROUP BY b, c HAVING udf(b) = 3 ORDER BY udf(b), udf(c) | ||
| -- !query 12 schema | ||
| struct<CAST(udf(cast(b as string)) AS INT):int,CAST(udf(cast(c as string)) AS STRING):string> | ||
| -- !query 12 output | ||
| 3 BBBB | ||
| 3 bbbb | ||
|
|
||
|
|
||
| -- !query 13 | ||
| SELECT udf(c), max(udf(a)) FROM test_having | ||
| GROUP BY c HAVING udf(count(*)) > 2 OR udf(min(a)) = udf(max(a)) | ||
| ORDER BY c | ||
| -- !query 13 schema | ||
| struct<CAST(udf(cast(c as string)) AS STRING):string,max(CAST(udf(cast(a as string)) AS INT)):int> | ||
| -- !query 13 output | ||
| XXXX 0 | ||
| bbbb 5 | ||
|
|
||
|
|
||
| -- !query 14 | ||
| SELECT udf(udf(min(udf(a)))), udf(udf(max(udf(a)))) FROM test_having HAVING udf(udf(min(udf(a)))) = udf(udf(max(udf(a)))) | ||
| -- !query 14 schema | ||
| struct<CAST(udf(cast(cast(udf(cast(min(cast(udf(cast(a as string)) as int)) as string)) as int) as string)) AS INT):int,CAST(udf(cast(cast(udf(cast(max(cast(udf(cast(a as string)) as int)) as string)) as int) as string)) AS INT):int> | ||
| -- !query 14 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 15 | ||
| SELECT udf(min(udf(a))), udf(udf(max(a))) FROM test_having HAVING udf(min(a)) < udf(max(udf(a))) | ||
| -- !query 15 schema | ||
| struct<CAST(udf(cast(min(cast(udf(cast(a as string)) as int)) as string)) AS INT):int,CAST(udf(cast(cast(udf(cast(max(a) as string)) as int) as string)) AS INT):int> | ||
| -- !query 15 output | ||
| 0 9 | ||
|
|
||
|
|
||
| -- !query 16 | ||
| SELECT udf(a) FROM test_having HAVING udf(min(a)) < udf(max(a)) | ||
| -- !query 16 schema | ||
| struct<> | ||
| -- !query 16 output | ||
| org.apache.spark.sql.AnalysisException | ||
| grouping expressions sequence is empty, and 'default.test_having.`a`' is not an aggregate function. Wrap '(min(default.test_having.`a`) AS `min(a#x)`, max(default.test_having.`a`) AS `max(a#x)`)' in windowing function(s) or wrap 'default.test_having.`a`' in first() (or first_value) if you don't care which value you get.; | ||
|
|
||
|
|
||
| -- !query 17 | ||
| SELECT 1 AS one FROM test_having HAVING udf(a) > 1 | ||
| -- !query 17 schema | ||
| struct<> | ||
| -- !query 17 output | ||
| org.apache.spark.sql.AnalysisException | ||
| cannot resolve '`a`' given input columns: [one]; line 1 pos 44 | ||
|
|
||
|
|
||
| -- !query 18 | ||
| SELECT 1 AS one FROM test_having HAVING udf(udf(1) > udf(2)) | ||
| -- !query 18 schema | ||
| struct<one:int> | ||
| -- !query 18 output | ||
|
|
||
|
|
||
|
|
||
| -- !query 19 | ||
| SELECT 1 AS one FROM test_having HAVING udf(udf(1) < udf(2)) | ||
| -- !query 19 schema | ||
| struct<one:int> | ||
| -- !query 19 output | ||
| 1 | ||
|
|
||
|
|
||
| -- !query 20 | ||
| SELECT 1 AS one FROM test_having WHERE 1/udf(a) = 1 HAVING 1 < 2 | ||
| -- !query 20 schema | ||
| struct<one:int> | ||
| -- !query 20 output | ||
| 1 | ||
|
|
||
|
|
||
| -- !query 21 | ||
| DROP TABLE test_having | ||
| -- !query 21 schema | ||
| struct<> | ||
| -- !query 21 output | ||
|
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.