[SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compile Spark REPL for Scala 2.12 + other 2.12 fixes #19307
Conversation
There was a problem hiding this comment.
These changes were necessary to make the implicits work in 2.12 now that eta-expansion of zero-arg methods is deprecated and apparently doesn't work for implicit resolution. It passes MiMa, but we could be conservative and retain the existing methods, and add function vals instead.
There was a problem hiding this comment.
ClassTag was required here, for reasons I don't fully get
There was a problem hiding this comment.
These changes avoid warnings about eta-expansion of zero-arg methods. It works fine in 2.11 as well; just not relying on syntactic sugar for the same.
There was a problem hiding this comment.
onFailure, onSuccess are deprecated in 2.12. This should be equivalent in 2.11+2.12
There was a problem hiding this comment.
Stack is deprecated in 2.12 for poor performance; ArrayStack should work the same and be faster, in 2.11 too
external/kafka-0-10-sql/pom.xml
Outdated
There was a problem hiding this comment.
Only 0.10.1+ supports Scala 2.12. By the time a 2.12 build is actually supported we may be on to 0.10.2. Not sure. This at least makes it work
There was a problem hiding this comment.
This is the only meaningful difference from the 2.11 REPL, as it has to hook into a different place. All other REPL-related code isn't specific to 2.11/2.12 and was moved out into the common src directory in the repl module
There was a problem hiding this comment.
resNaN1 is an Array[Double] so can never contain null. This is always true and generated a warning
There was a problem hiding this comment.
This whole file fix is already going into master separately as a hotfix
sql/hive-thriftserver/pom.xml
Outdated
There was a problem hiding this comment.
For some reason, hive-thriftserver wouldn't compile, unable to find jetty.server classes. It has a point, these should explicit, even if it somehow worked before.
|
Test build #82033 has finished for PR 19307 at commit
|
|
retest this please |
|
Test build #82036 has finished for PR 19307 at commit
|
|
Good to see this passes, as I expected, because it means we can also commit this chunk of progress towards 2.12 without any downside for 2.11. Right now this means that the 2.12 build does compile, but I know it doesn't quite pass all tests (some weird paranamer problem, to start with) The only part I was slightly uncertain about was the change to sequenceFile implicits, but MiMa hasn't complained. Worth a look at the changes, but I think this is worth committing before too long. |
|
Test build #82076 has finished for PR 19307 at commit
|
|
Merged to master |
3 participants
What changes were proposed in this pull request?
Enable Scala 2.12 REPL. Fix most remaining issues with 2.12 compilation and warnings, including:
How was this patch tested?
Existing tests