[SPARK-25307][SQL] ArraySort function may return an error in the code generation phase #22314

dilipbiswal · 2018-09-02T04:07:28Z

What changes were proposed in this pull request?

Sorting array of booleans (not nullable) returns a compilation error in the code generation phase. Below is the compilation error :

java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 51, Column 23: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 51, Column 23: No applicable constructor/method found for actual parameters "boolean[]"; candidates are: "public static void java.util.Arrays.sort(long[])", "public static void java.util.Arrays.sort(long[], int, int)", "public static void java.util.Arrays.sort(byte[], int, int)", "public static void java.util.Arrays.sort(float[])", "public static void java.util.Arrays.sort(float[], int, int)", "public static void java.util.Arrays.sort(char[])", "public static void java.util.Arrays.sort(char[], int, int)", "public static void java.util.Arrays.sort(short[], int, int)", "public static void java.util.Arrays.sort(short[])", "public static void java.util.Arrays.sort(byte[])", "public static void java.util.Arrays.sort(java.lang.Object[], int, int, java.util.Comparator)", "public static void java.util.Arrays.sort(java.lang.Object[], java.util.Comparator)", "public static void java.util.Arrays.sort(int[])", "public static void java.util.Arrays.sort(java.lang.Object[], int, int)", "public static void java.util.Arrays.sort(java.lang.Object[])", "public static void java.util.Arrays.sort(double[])", "public static void java.util.Arrays.sort(double[], int, int)", "public static void java.util.Arrays.sort(int[], int, int)"
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
	at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2380)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
	at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:1305)

How was this patch tested?

Added test in collectionExpressionSuite

kiszk · 2018-09-02T05:01:13Z

Good catch, LGTM

dilipbiswal · 2018-09-02T05:55:45Z

@kiszk Thanks. Can you please help look at 22315 as well.

SparkQA · 2018-09-02T07:05:02Z

Test build #95585 has finished for PR 22314 at commit 7dec581.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2018-09-02T07:06:51Z

retest this please

ueshin · 2018-09-02T10:01:58Z

LGTM.

SparkQA · 2018-09-02T11:05:17Z

Test build #95588 has finished for PR 22314 at commit 7dec581.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2018-09-03T02:30:26Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

nonNullPrimitiveNumericAscendingSort or fastSortCode preferred?

@maropu Lets keep the variable name the same. I personally find the name more descriptive. Lets add a private def like you suggest in the following comment.

maropu · 2018-09-03T02:31:18Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

How about making this condition a separate private method like isFastSortSupported?

will make a change

maropu · 2018-09-03T06:23:43Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

sorry, but not method but variable is ok (I said method in the previous comment though...), e.g.,

val canPerformFastSort = { CodeGenerator.isPrimitiveType(elementType) && elementType != BooleanType && !containsNull }

SparkQA · 2018-09-03T07:05:01Z

Test build #95603 has finished for PR 22314 at commit 2f1b192.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-09-03T07:05:01Z

Test build #95601 has finished for PR 22314 at commit 2e893ad.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2018-09-03T07:09:23Z

retest this please

SparkQA · 2018-09-03T07:57:54Z

Test build #95604 has finished for PR 22314 at commit 2f1b192.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…ration phase.

SparkQA · 2018-09-03T13:01:02Z

Test build #95617 has finished for PR 22314 at commit 3645438.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-09-03T21:17:18Z

Test build #95635 has finished for PR 22314 at commit d27256e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2018-09-04T04:39:02Z

Thanks! merging to master.

dilipbiswal · 2018-09-04T04:43:56Z

@ueshin @kiszk @maropu Thanks a lot.

ueshin · 2018-09-04T04:49:47Z

@dilipbiswal Do we need to backport this to 2.3? If so, could you submit a backport pr to branch-2.3 please? Thanks!

dilipbiswal · 2018-09-04T04:53:20Z

@ueshin Sure.

dilipbiswal · 2018-09-04T06:01:43Z

@ueshin We don't have codegen enabled for this function in 2.3. So thankfully we don't have this problem :-)

ueshin · 2018-09-04T07:02:16Z

@dilipbiswal Thanks for confirming!

dilipbiswal changed the title ~~[SPARK-25307] ArraySort function may return an error in the code generation phase~~ [SPARK-25307][SQL] ArraySort function may return an error in the code generation phase Sep 2, 2018

dilipbiswal mentioned this pull request Sep 2, 2018

[SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive #22270

Closed

maropu reviewed Sep 3, 2018

View reviewed changes

dilipbiswal added 3 commits September 3, 2018 01:55

[SPARK-25307] ArraySort function may return an error in the code gene…

8c07d3c

…ration phase.

Code review

1547ac7

fix

3645438

dilipbiswal force-pushed the SPARK-25307 branch from 2f1b192 to 3645438 Compare September 3, 2018 08:56

minor

d27256e

asfgit closed this in b60ee3a Sep 4, 2018

[SPARK-25307][SQL] ArraySort function may return an error in the code generation phase #22314

[SPARK-25307][SQL] ArraySort function may return an error in the code generation phase #22314

Uh oh!

Conversation

dilipbiswal commented Sep 2, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

kiszk commented Sep 2, 2018

Uh oh!

dilipbiswal commented Sep 2, 2018

Uh oh!

SparkQA commented Sep 2, 2018

Uh oh!

dilipbiswal commented Sep 2, 2018

Uh oh!

ueshin commented Sep 2, 2018

Uh oh!

SparkQA commented Sep 2, 2018

Uh oh!

maropu Sep 3, 2018

Choose a reason for hiding this comment

Uh oh!

dilipbiswal Sep 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maropu Sep 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dilipbiswal Sep 3, 2018

Choose a reason for hiding this comment

Uh oh!

maropu Sep 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 3, 2018

Uh oh!

SparkQA commented Sep 3, 2018

Uh oh!

dilipbiswal commented Sep 3, 2018

Uh oh!

SparkQA commented Sep 3, 2018

Uh oh!

SparkQA commented Sep 3, 2018

Uh oh!

SparkQA commented Sep 3, 2018

Uh oh!

ueshin commented Sep 4, 2018

Uh oh!

dilipbiswal commented Sep 4, 2018

Uh oh!

ueshin commented Sep 4, 2018

Uh oh!

dilipbiswal commented Sep 4, 2018

Uh oh!

dilipbiswal commented Sep 4, 2018

Uh oh!

ueshin commented Sep 4, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dilipbiswal Sep 3, 2018 •

edited

Loading

maropu Sep 3, 2018 •

edited

Loading

maropu Sep 3, 2018 •

edited

Loading