[SPARK-29855][SQL] typed literals with negative sign with proper result or exception #26479

yaooqinn · 2019-11-12T05:53:01Z

What changes were proposed in this pull request?

-- !query 83
select -integer '7'
-- !query 83 schema
struct<7:int>
-- !query 83 output
7

-- !query 86
select -date '1999-01-01'
-- !query 86 schema
struct<DATE '1999-01-01':date>
-- !query 86 output
1999-01-01

-- !query 87
select -timestamp '1999-01-01'
-- !query 87 schema
struct<TIMESTAMP('1999-01-01 00:00:00'):timestamp>
-- !query 87 output
1999-01-01 00:00:00

the integer should be -7 and the date and timestamp results are confusing which should throw exceptions

Why are the changes needed?

bug fix

Does this PR introduce any user-facing change?

NO

How was this patch tested?

ADD UTs

…lt or exception

yaooqinn · 2019-11-12T05:55:35Z

cc @cloud-fan @maropu @HyukjinKwon @MaxGekk, thanks a lot

cloud-fan · 2019-11-12T06:54:33Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

    val value = string(ctx.STRING)
    val valueType = ctx.identifier.getText.toUpperCase(Locale.ROOT)
+    val negativeSign = Option(ctx.negativeSign).map(_.getText).getOrElse("")
+    val isNegative = if (negativeSign == "-") true else false


is it simply val isNegative = ctx.negativeSign != null?

cloud-fan · 2019-11-12T06:55:19Z

looks reasonable to me. Is it a common behavior in other DBs like pgsql?

MaxGekk · 2019-11-12T06:54:47Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

    val value = string(ctx.STRING)
    val valueType = ctx.identifier.getText.toUpperCase(Locale.ROOT)
+    val negativeSign = Option(ctx.negativeSign).map(_.getText).getOrElse("")
+    val isNegative = if (negativeSign == "-") true else false


Suggested change

val isNegative = if (negativeSign == "-") true else false

val isNegative = ctx.negativeSign != null && ctx.negativeSign.getText == "-"

and remove val negativeSign

do we really need to check ctx.negativeSign.getText? the parser rule is MINUS? ...

negativeSign is used in error message, but I think @cloud-fan 's point is right, we need not to check ctx.negativeSign.getText. I'd do some tests to verify this.

SparkQA · 2019-11-12T08:05:02Z

Test build #113615 has finished for PR 26479 at commit 1d04cb3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2019-11-12T08:06:59Z

looks reasonable to me. Is it a common behavior in other DBs like pgsql?

yes it right the pg behavior

cloud-fan · 2019-11-12T09:11:08Z

sql/core/src/test/resources/sql-tests/inputs/literals.sql

 select '1' year to second;
 select interval 1 year '2-1' year to month;
 select 1 year '2-1' year to month;
+SET spark.sql.ansi.enabled=false;


not related to this PR, but it's really annoying to duplicate the test cases with ansi mode on and off.

I'm wondering if we can introduce a -- import abc.sql directive so that we can easily import test cases in ansi/interval.sql, with several ansi specific test cases. cc @maropu

Ah, I see. That's certainly annoying. I'll try to make a pr later for that approach.

SparkQA · 2019-11-12T12:53:18Z

Test build #113617 has finished for PR 26479 at commit 8f524c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-11-12T14:52:50Z

Merged to master.

…s from another test case in SQLQueryTestSuite ### What changes were proposed in this pull request? This pr is to support `--import` directive to load queries from another test case in SQLQueryTestSuite. This fix comes from the cloud-fan suggestion in #26479 (comment) ### Why are the changes needed? This functionality might reduce duplicate test code in `SQLQueryTestSuite`. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Run `SQLQueryTestSuite`. Closes #26497 from maropu/ImportTests. Authored-by: Takeshi Yamamuro <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

[SPARK-29855][SQL] typed literals with negative sign with proper resu…

1d04cb3

…lt or exception

cloud-fan reviewed Nov 12, 2019

View reviewed changes

MaxGekk reviewed Nov 12, 2019

View reviewed changes

simplify negative check

8f524c3

cloud-fan reviewed Nov 12, 2019

View reviewed changes

cloud-fan approved these changes Nov 12, 2019

View reviewed changes

HyukjinKwon approved these changes Nov 12, 2019

View reviewed changes

HyukjinKwon closed this in d99398e Nov 12, 2019

maropu mentioned this pull request Nov 13, 2019

[SPARK-29873][SQL][TESTS] Support --import directive to load queries from another test case in SQLQueryTestSuite #26497

Closed

dongjoon-hyun added the SQL label Feb 5, 2020

	val isNegative = if (negativeSign == "-") true else false
	val isNegative = ctx.negativeSign != null && ctx.negativeSign.getText == "-"

[SPARK-29855][SQL] typed literals with negative sign with proper result or exception #26479

[SPARK-29855][SQL] typed literals with negative sign with proper result or exception #26479

Uh oh!

Conversation

yaooqinn commented Nov 12, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

yaooqinn commented Nov 12, 2019

Uh oh!

cloud-fan Nov 12, 2019

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Nov 12, 2019

Uh oh!

MaxGekk Nov 12, 2019

Choose a reason for hiding this comment

Uh oh!

MaxGekk Nov 12, 2019

Choose a reason for hiding this comment

Uh oh!

cloud-fan Nov 12, 2019

Choose a reason for hiding this comment

Uh oh!

yaooqinn Nov 12, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 12, 2019

Uh oh!

yaooqinn commented Nov 12, 2019

Uh oh!

cloud-fan Nov 12, 2019

Choose a reason for hiding this comment

Uh oh!

maropu Nov 13, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 12, 2019

Uh oh!

HyukjinKwon commented Nov 12, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants