[SPARK-18950][SQL] Report conflicting fields when merging two StructTypes #16365

jiayue-zhang · 2016-12-21T00:00:25Z

What changes were proposed in this pull request?

Currently, StructType.merge() only reports data types of conflicting fields when merging two incompatible schemas. It would be nice to also report the field names for easier debugging.

How was this patch tested?

Unit test in DataTypeSuite.
Print exception message when conflict is triggered.

…ypes

HyukjinKwon · 2016-12-21T01:21:19Z

This is actually the message users face in some cases. Isn't it :)?

val df1 = spark.range(10).selectExpr("id as intcol", "cast(id as int) as longcol")
df1.write.parquet("/tmp/a")
val df2 = spark.range(10).selectExpr("id as intcol", "id as longcol")
df2.write.parquet("/tmp/b")
spark.read.option("mergeSchema", true).parquet("/tmp/a", "/tmp/b").show()

Before

...
Caused by: org.apache.spark.SparkException: Failed to merge incompatible data types IntegerType and LongType
  at org.apache.spark.sql.types.StructType$.merge(StructType.scala:515)
  at org.apache.spark.sql.types.StructType$$anonfun$merge$1$$anonfun$apply$3.apply(StructType.scala:473)
  at org.apache.spark.sql.types.StructType$$anonfun$merge$1$$anonfun$apply$3.apply(StructType.scala:471)
...

After

...
Caused by: org.apache.spark.SparkException: Failed to merge field longcol: Failed to merge incompatible data types IntegerType and LongType
  at org.apache.spark.sql.types.StructType$$anonfun$merge$1$$anonfun$apply$3.apply(StructType.scala:480)
  at org.apache.spark.sql.types.StructType$$anonfun$merge$1$$anonfun$apply$3.apply(StructType.scala:471)
  at scala.Option.map(Option.scala:146)
...

BTW, it looks the test in DataTypeSuite was not added by mistake.

jiayue-zhang · 2016-12-21T02:12:48Z

Thanks for the review @HyukjinKwon !
Are your stacktrace of Before and After swapped?
Do you mean the message is confusing? How about correcting it to Failed to merge field longcol: incompatible data types IntegerType and LongType?
For the missing test case, do you mean a merge case that are not StructType as in your example?
Or is it something else that I'm missing?

HyukjinKwon · 2016-12-21T02:42:37Z

Doh, yeap, I just swapped back. I just simply meant I support this PR because this improves user's experience with a better message as well :).

gatorsmile · 2017-06-13T06:26:20Z

sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala

+                      dataType = dataType,
+                      nullable = leftNullable || rightNullable)
+                  case Failure(e) =>
+                    throw new SparkException(s"Failed to merge field $leftName: " + e.getMessage)


$leftName -> '$leftName'

gatorsmile · 2017-06-13T06:26:27Z

ok to test

gatorsmile · 2017-06-13T06:26:57Z

@bravo-zhang Sorry for the late response. Could you please also add a test case for capturing the new error message?

SparkQA · 2017-06-13T06:27:37Z

Test build #77970 has started for PR 16365 at commit 9bc9f8a.

gatorsmile · 2017-06-13T17:13:40Z

retest this please

SparkQA · 2017-06-13T19:30:46Z

Test build #77999 has finished for PR 16365 at commit 9bc9f8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-07-24T03:32:34Z

ping @bravo-zhang for adding the test.

SparkQA · 2017-07-25T02:07:02Z

Test build #79921 has finished for PR 16365 at commit 54a898f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiayue-zhang · 2017-07-25T16:29:17Z

@HyukjinKwon @gatorsmile Test added.

gatorsmile · 2017-07-31T01:59:52Z

retest this please

gatorsmile · 2017-07-31T02:05:56Z

sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala

-                leftField.copy(
-                  dataType = merge(leftType, rightType),
-                  nullable = leftNullable || rightNullable)
+                Try {


Could we use the JAVA style try and catch? See the https://github.com/databricks/scala-style-guide#exception-handling-try-vs-try

gatorsmile · 2017-07-31T02:07:08Z

sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala

+                      dataType = dataType,
+                      nullable = leftNullable || rightNullable)
+                  case Failure(e) =>
+                    throw new SparkException(s"Failed to merge field '$leftName': " + e.getMessage)


Could we throw an AnalysisException with both sides, left and right? Thanks!

Other exceptions in this class are also SparkException, for example the precision conflicts. Should we keep it as SparkException?
For "with both sides, left and right", do you mean just to modify the message a bit to include both left and right names(though they are the same)?
@gatorsmile your other comments are resolved.

gatorsmile · 2017-07-31T02:07:39Z

sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala

      left.merge(right)
-    }
+    }.getMessage
+    assert(message.contains("conflictColumn"))


Could we capture the whole message? It can help us review the error message.

SparkQA · 2017-07-31T04:33:30Z

Test build #80064 has finished for PR 16365 at commit 54a898f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-31T18:49:04Z

Test build #80088 has finished for PR 16365 at commit f500a11.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-31T21:24:34Z

Test build #80090 has finished for PR 16365 at commit f4892c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-08-01T00:21:29Z

LGTM

Thanks! Merging to master.

[SPARK-18950][SQL] Report conflicting fields when merging two StructT…

9bc9f8a

…ypes

gatorsmile reviewed Jun 13, 2017

View reviewed changes

Add test for merge conflict message.

54a898f

gatorsmile reviewed Jul 31, 2017

View reviewed changes

JAVA try catch and test whole error message.

f500a11

Fix scala style check

f4892c9

asfgit closed this in 6b186c9 Aug 1, 2017

jiayue-zhang deleted the spark-18950 branch August 2, 2017 16:09

[SPARK-18950][SQL] Report conflicting fields when merging two StructTypes #16365

[SPARK-18950][SQL] Report conflicting fields when merging two StructTypes #16365

Uh oh!

Conversation

jiayue-zhang commented Dec 21, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon commented Dec 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiayue-zhang commented Dec 21, 2016

Uh oh!

HyukjinKwon commented Dec 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile Jun 13, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jun 13, 2017

Uh oh!

gatorsmile commented Jun 13, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

gatorsmile commented Jun 13, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

HyukjinKwon commented Jul 24, 2017

Uh oh!

SparkQA commented Jul 25, 2017

Uh oh!

jiayue-zhang commented Jul 25, 2017

Uh oh!

gatorsmile commented Jul 31, 2017

Uh oh!

gatorsmile Jul 31, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile Jul 31, 2017

Choose a reason for hiding this comment

Uh oh!

jiayue-zhang Jul 31, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile Jul 31, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 31, 2017

Uh oh!

SparkQA commented Jul 31, 2017

Uh oh!

SparkQA commented Jul 31, 2017

Uh oh!

gatorsmile commented Aug 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HyukjinKwon commented Dec 21, 2016 •

edited

Loading

HyukjinKwon commented Dec 21, 2016 •

edited

Loading

gatorsmile commented Aug 1, 2017 •

edited

Loading