[SPARK-31489][SQL] Fix pushing down filters with `java.time.LocalDate` values in ORC #28261

MaxGekk · 2020-04-19T17:53:15Z

What changes were proposed in this pull request?

Convert java.time.LocalDate to java.sql.Date in pushed down filters to ORC datasource when Java 8 time API enabled.

Why are the changes needed?

The changes fix the exception raised while pushing date filters when spark.sql.datetime.java8API.enabled is set to true:

Wrong value class java.time.LocalDate for DATE.EQUALS leaf
java.lang.IllegalArgumentException: Wrong value class java.time.LocalDate for DATE.EQUALS leaf
	at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.checkLiteralType(SearchArgumentImpl.java:192)
	at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.<init>(SearchArgumentImpl.java:75)
	at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$BuilderImpl.equals(SearchArgumentImpl.java:352)
	at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.buildLeafSearchArgument(OrcFilters.scala:229)

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

Added tests to OrcFilterSuite.

SparkQA · 2020-04-19T19:18:58Z

Test build #121482 has finished for PR 28261 at commit 392c0e6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-04-19T19:39:34Z

jenkins, retest this, please

SparkQA · 2020-04-20T00:20:29Z

Test build #121485 has finished for PR 28261 at commit 392c0e6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-04-20T06:13:14Z

@dongjoon-hyun @cloud-fan @HyukjinKwon Could you take a look at the PR, please.

cloud-fan · 2020-04-20T07:24:13Z

...core/v2.3/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala

+    val dates = Seq("2017-08-18", "2017-08-19", "2017-08-20", "2017-08-21").map { day =>
+      LocalDate.parse(day)
+    }
+    withSQLConf(SQLConf.DATETIME_JAVA8API_ENABLED.key -> "true") {


It makes me realize that this config can break data source v1/v2 as it changes the value in Filter.

Shall we fix DataSourceAnalysis.translateLeafNodeFilter instead to always return java 7 Date/Timestamp?

@cloud-fan Here is the PR #28272

@dongjoon-hyun ^^ The PR covers this PR

Got it. Thanks, @MaxGekk !

dongjoon-hyun · 2020-04-20T10:20:21Z

Thank you for pinging me, @MaxGekk .
Could you handle the following together?

sql/core/v1.2/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
sql/core/v1.2/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala

dongjoon-hyun · 2020-04-20T10:22:00Z

Retest this please.

dongjoon-hyun · 2020-04-20T10:23:09Z

...core/v2.3/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala

    }
  }
+
+  test("filter pushdown - local date") {


nit. test("SPARK-31489: filter.

dongjoon-hyun · 2020-04-20T10:24:12Z

For this PR, please pass the PR builders with both Hive 1.2 and Hive 2.3.

SparkQA · 2020-04-20T15:37:56Z

Test build #121523 has finished for PR 28261 at commit 392c0e6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-04-23T15:58:33Z

jenkins, retest this, please

…lter-pushdown

SparkQA · 2020-04-23T21:02:54Z

Test build #121696 has finished for PR 28261 at commit e5952e1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-23T22:05:33Z

Test build #121685 has finished for PR 28261 at commit 392c0e6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-04-24T05:07:43Z

retest this please

SparkQA · 2020-04-24T07:05:01Z

Test build #121719 has finished for PR 28261 at commit e5952e1.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-04-24T07:17:45Z

jenkins, retest this, please

SparkQA · 2020-04-24T11:13:52Z

Test build #121735 has finished for PR 28261 at commit e5952e1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-04-24T11:25:08Z

jenkins, retest this, please

SparkQA · 2020-04-24T17:44:52Z

Test build #121754 has finished for PR 28261 at commit e5952e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-04-25T20:23:09Z

jenkins, retest this, please

SparkQA · 2020-04-25T21:55:53Z

Test build #121813 has finished for PR 28261 at commit e5952e1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-04-26T06:13:58Z

jenkins, retest this, please

SparkQA · 2020-04-26T07:05:01Z

Test build #121833 has finished for PR 28261 at commit e5952e1.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-26T11:47:17Z

Test build #121836 has finished for PR 28261 at commit aa57be0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-04-26T14:00:07Z

jenkins, retest this, please

SparkQA · 2020-04-26T18:19:01Z

Test build #121847 has finished for PR 28261 at commit aa57be0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-04-26T18:24:41Z

This PR has been tested for hive-1.2 and hive-2.3, and it is ready for review. @cloud-fan @dongjoon-hyun Please, take a look at it.

dongjoon-hyun

+1, LGTM. Thank you, @MaxGekk and @cloud-fan .
Merged to master/3.0.

…` values in ORC ### What changes were proposed in this pull request? Convert `java.time.LocalDate` to `java.sql.Date` in pushed down filters to ORC datasource when Java 8 time API enabled. Closes #28272 ### Why are the changes needed? The changes fix the exception raised while pushing date filters when `spark.sql.datetime.java8API.enabled` is set to `true`: ``` Wrong value class java.time.LocalDate for DATE.EQUALS leaf java.lang.IllegalArgumentException: Wrong value class java.time.LocalDate for DATE.EQUALS leaf at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.checkLiteralType(SearchArgumentImpl.java:192) at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.<init>(SearchArgumentImpl.java:75) at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$BuilderImpl.equals(SearchArgumentImpl.java:352) at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.buildLeafSearchArgument(OrcFilters.scala:229) ``` ### Does this PR introduce any user-facing change? Yes ### How was this patch tested? Added tests to `OrcFilterSuite`. Closes #28261 from MaxGekk/orc-date-filter-pushdown. Authored-by: Max Gekk <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit bd139bd) Signed-off-by: Dongjoon Hyun <[email protected]>

MaxGekk added 3 commits April 19, 2020 20:46

Add tests

71bb4cb

Fix: convert to java.sql.Date

cde48ee

Add implicit conversion of java.time.LocalDate to an expression

392c0e6

probot-autolabeler bot added the SQL label Apr 19, 2020

cloud-fan reviewed Apr 20, 2020

View reviewed changes

cloud-fan mentioned this pull request Apr 20, 2020

[SPARK-31488][SQL] Support java.time.LocalDate in Parquet filter pushdown #28259

Closed

MaxGekk mentioned this pull request Apr 20, 2020

[SPARK-31489][SPARK-31488][SQL] Translate date values of pushed down filters to java.sql.Date #28272

Closed

dongjoon-hyun changed the title ~~[SPARK-31489][SQL] Fix pushing down filters with java.time.LocalDate values in ORC~~ [SPARK-31489][SQL][test-hive1.2] Fix pushing down filters with java.time.LocalDate values in ORC Apr 20, 2020

dongjoon-hyun reviewed Apr 20, 2020

View reviewed changes

MaxGekk added 2 commits April 23, 2020 21:07

Merge remote-tracking branch 'remotes/origin/master' into orc-date-fi…

ae18964

…lter-pushdown

Dedup tests

e5952e1

MaxGekk changed the title ~~[SPARK-31489][SQL][test-hive1.2] Fix pushing down filters with java.time.LocalDate values in ORC~~ [SPARK-31489][SQL] Fix pushing down filters with java.time.LocalDate values in ORC Apr 23, 2020

MaxGekk changed the title ~~[SPARK-31489][SQL] Fix pushing down filters with java.time.LocalDate values in ORC~~ [SPARK-31489][SQL][test-hive1.2] Fix pushing down filters with java.time.LocalDate values in ORC Apr 25, 2020

Fix for hive 1.2

aa57be0

MaxGekk changed the title ~~[SPARK-31489][SQL][test-hive1.2] Fix pushing down filters with java.time.LocalDate values in ORC~~ [SPARK-31489][SQL] Fix pushing down filters with java.time.LocalDate values in ORC Apr 26, 2020

dongjoon-hyun approved these changes Apr 26, 2020

View reviewed changes

dongjoon-hyun closed this in bd139bd Apr 26, 2020

MaxGekk mentioned this pull request May 25, 2020

[SPARK-31818][SQL] Fix pushing down filters with java.time.Instant values in ORC #28636

Closed

MaxGekk deleted the orc-date-filter-pushdown branch June 5, 2020 19:48

[SPARK-31489][SQL] Fix pushing down filters with java.time.LocalDate values in ORC #28261

[SPARK-31489][SQL] Fix pushing down filters with java.time.LocalDate values in ORC #28261

Uh oh!

Conversation

MaxGekk commented Apr 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Apr 19, 2020

Uh oh!

MaxGekk commented Apr 19, 2020

Uh oh!

SparkQA commented Apr 20, 2020

Uh oh!

MaxGekk commented Apr 20, 2020

Uh oh!

cloud-fan Apr 20, 2020

Choose a reason for hiding this comment

Uh oh!

MaxGekk Apr 20, 2020

Choose a reason for hiding this comment

Uh oh!

MaxGekk Apr 20, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Apr 20, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Apr 20, 2020

Uh oh!

dongjoon-hyun Apr 20, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 20, 2020

Uh oh!

SparkQA commented Apr 20, 2020

Uh oh!

MaxGekk commented Apr 23, 2020

Uh oh!

SparkQA commented Apr 23, 2020

Uh oh!

SparkQA commented Apr 23, 2020

Uh oh!

cloud-fan commented Apr 24, 2020

Uh oh!

SparkQA commented Apr 24, 2020

Uh oh!

MaxGekk commented Apr 24, 2020

Uh oh!

SparkQA commented Apr 24, 2020

Uh oh!

MaxGekk commented Apr 24, 2020

Uh oh!

SparkQA commented Apr 24, 2020

Uh oh!

MaxGekk commented Apr 25, 2020

Uh oh!

SparkQA commented Apr 25, 2020

Uh oh!

MaxGekk commented Apr 26, 2020

Uh oh!

SparkQA commented Apr 26, 2020

Uh oh!

SparkQA commented Apr 26, 2020

Uh oh!

MaxGekk commented Apr 26, 2020

Uh oh!

SparkQA commented Apr 26, 2020

Uh oh!

MaxGekk commented Apr 26, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

[SPARK-31489][SQL] Fix pushing down filters with `java.time.LocalDate` values in ORC #28261

[SPARK-31489][SQL] Fix pushing down filters with `java.time.LocalDate` values in ORC #28261

MaxGekk commented Apr 19, 2020 •

edited

Loading

dongjoon-hyun commented Apr 20, 2020 •

edited

Loading