Skip to content

[SPARK-25956] Make Scala 2.12 as default Scala version in Spark 3.0 #22967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dbtsai wants to merge 3 commits into from
Closed

[SPARK-25956] Make Scala 2.12 as default Scala version in Spark 3.0 #22967

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
52dc4a1
make scala 2.12 as default
dbtsai Nov 6, 2018
a5cc336
address feedback
dbtsai Nov 14, 2018
8b6b474
fix python tests
dbtsai Nov 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>spark-assembly_2.11</artifactId>
<artifactId>spark-assembly_2.12</artifactId>
<name>Spark Project Assembly</name>
<url>http://spark.apache.org/</url>
<packaging>pom</packaging>
Expand Down
4 changes: 2 additions & 2 deletions common/kvstore/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>spark-kvstore_2.11</artifactId>
<artifactId>spark-kvstore_2.12</artifactId>
<packaging>jar</packaging>
<name>Spark Project Local DB</name>
<url>http://spark.apache.org/</url>
Expand Down
4 changes: 2 additions & 2 deletions common/network-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>spark-network-common_2.11</artifactId>
<artifactId>spark-network-common_2.12</artifactId>
<packaging>jar</packaging>
<name>Spark Project Networking</name>
<url>http://spark.apache.org/</url>
Expand Down
4 changes: 2 additions & 2 deletions common/network-shuffle/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>spark-network-shuffle_2.11</artifactId>
<artifactId>spark-network-shuffle_2.12</artifactId>
<packaging>jar</packaging>
<name>Spark Project Shuffle Streaming Service</name>
<url>http://spark.apache.org/</url>
Expand Down
4 changes: 2 additions & 2 deletions common/network-yarn/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>spark-network-yarn_2.11</artifactId>
<artifactId>spark-network-yarn_2.12</artifactId>
<packaging>jar</packaging>
<name>Spark Project YARN Shuffle Service</name>
<url>http://spark.apache.org/</url>
Expand Down
4 changes: 2 additions & 2 deletions common/sketch/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>spark-sketch_2.11</artifactId>
<artifactId>spark-sketch_2.12</artifactId>
<packaging>jar</packaging>
<name>Spark Project Sketch</name>
<url>http://spark.apache.org/</url>
Expand Down
4 changes: 2 additions & 2 deletions common/tags/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>spark-tags_2.11</artifactId>
<artifactId>spark-tags_2.12</artifactId>
<packaging>jar</packaging>
<name>Spark Project Tags</name>
<url>http://spark.apache.org/</url>
Expand Down
4 changes: 2 additions & 2 deletions common/unsafe/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>spark-unsafe_2.11</artifactId>
<artifactId>spark-unsafe_2.12</artifactId>
<packaging>jar</packaging>
<name>Spark Project Unsafe</name>
<url>http://spark.apache.org/</url>
Expand Down
4 changes: 2 additions & 2 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>spark-core_2.11</artifactId>
<artifactId>spark-core_2.12</artifactId>
<properties>
<sbt.project.name>core</sbt.project.name>
</properties>
Expand Down
36 changes: 18 additions & 18 deletions dev/deps/spark-deps-hadoop-2.7
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,13 @@ avro-1.8.2.jar
avro-ipc-1.8.2.jar
avro-mapred-1.8.2-hadoop2.jar
bonecp-0.8.0.RELEASE.jar
breeze-macros_2.11-0.13.2.jar
breeze_2.11-0.13.2.jar
breeze-macros_2.12-0.13.2.jar
breeze_2.12-0.13.2.jar
calcite-avatica-1.2.0-incubating.jar
calcite-core-1.2.0-incubating.jar
calcite-linq4j-1.2.0-incubating.jar
chill-java-0.9.3.jar
chill_2.11-0.9.3.jar
chill_2.12-0.9.3.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
Expand Down Expand Up @@ -96,7 +96,7 @@ jackson-jaxrs-1.9.13.jar
jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations-2.9.6.jar
jackson-module-paranamer-2.9.6.jar
jackson-module-scala_2.11-2.9.6.jar
jackson-module-scala_2.12-2.9.6.jar
jackson-xc-1.9.13.jar
janino-3.0.10.jar
javassist-3.18.1-GA.jar
Expand All @@ -122,10 +122,10 @@ jline-2.14.6.jar
joda-time-2.9.3.jar
jodd-core-3.5.2.jar
jpam-1.1.jar
json4s-ast_2.11-3.5.3.jar
json4s-core_2.11-3.5.3.jar
json4s-jackson_2.11-3.5.3.jar
json4s-scalap_2.11-3.5.3.jar
json4s-ast_2.12-3.5.3.jar
json4s-core_2.12-3.5.3.jar
json4s-jackson_2.12-3.5.3.jar
json4s-scalap_2.12-3.5.3.jar
jsp-api-2.1.jar
jsr305-3.0.0.jar
jta-1.1.jar
Expand All @@ -140,8 +140,8 @@ libthrift-0.9.3.jar
log4j-1.2.17.jar
logging-interceptor-3.9.1.jar
lz4-java-1.5.0.jar
machinist_2.11-0.6.1.jar
macro-compat_2.11-1.1.1.jar
machinist_2.12-0.6.1.jar
macro-compat_2.12-1.1.1.jar
mesos-1.4.0-shaded-protobuf.jar
metrics-core-3.1.5.jar
metrics-graphite-3.1.5.jar
Expand Down Expand Up @@ -170,19 +170,19 @@ parquet-jackson-1.10.0.jar
protobuf-java-2.5.0.jar
py4j-0.10.8.1.jar
pyrolite-4.13.jar
scala-compiler-2.11.12.jar
scala-library-2.11.12.jar
scala-parser-combinators_2.11-1.1.0.jar
scala-reflect-2.11.12.jar
scala-xml_2.11-1.0.5.jar
shapeless_2.11-2.3.2.jar
scala-compiler-2.12.7.jar
scala-library-2.12.7.jar
scala-parser-combinators_2.12-1.1.0.jar
scala-reflect-2.12.7.jar
scala-xml_2.12-1.0.5.jar
shapeless_2.12-2.3.2.jar
slf4j-api-1.7.16.jar
slf4j-log4j12-1.7.16.jar
snakeyaml-1.18.jar
snappy-0.2.jar
snappy-java-1.1.7.1.jar
spire-macros_2.11-0.13.0.jar
spire_2.11-0.13.0.jar
spire-macros_2.12-0.13.0.jar
spire_2.12-0.13.0.jar
stax-api-1.0-2.jar
stax-api-1.0.1.jar
stream-2.7.0.jar
Expand Down
36 changes: 18 additions & 18 deletions dev/deps/spark-deps-hadoop-3.1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,13 @@ avro-1.8.2.jar
avro-ipc-1.8.2.jar
avro-mapred-1.8.2-hadoop2.jar
bonecp-0.8.0.RELEASE.jar
breeze-macros_2.11-0.13.2.jar
breeze_2.11-0.13.2.jar
breeze-macros_2.12-0.13.2.jar
breeze_2.12-0.13.2.jar
calcite-avatica-1.2.0-incubating.jar
calcite-core-1.2.0-incubating.jar
calcite-linq4j-1.2.0-incubating.jar
chill-java-0.9.3.jar
chill_2.11-0.9.3.jar
chill_2.12-0.9.3.jar
commons-beanutils-1.9.3.jar
commons-cli-1.2.jar
commons-codec-1.10.jar
Expand Down Expand Up @@ -96,7 +96,7 @@ jackson-jaxrs-json-provider-2.7.8.jar
jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations-2.9.6.jar
jackson-module-paranamer-2.9.6.jar
jackson-module-scala_2.11-2.9.6.jar
jackson-module-scala_2.12-2.9.6.jar
janino-3.0.10.jar
javassist-3.18.1-GA.jar
javax.annotation-api-1.2.jar
Expand All @@ -123,10 +123,10 @@ joda-time-2.9.3.jar
jodd-core-3.5.2.jar
jpam-1.1.jar
json-smart-2.3.jar
json4s-ast_2.11-3.5.3.jar
json4s-core_2.11-3.5.3.jar
json4s-jackson_2.11-3.5.3.jar
json4s-scalap_2.11-3.5.3.jar
json4s-ast_2.12-3.5.3.jar
json4s-core_2.12-3.5.3.jar
json4s-jackson_2.12-3.5.3.jar
json4s-scalap_2.12-3.5.3.jar
jsp-api-2.1.jar
jsr305-3.0.0.jar
jta-1.1.jar
Expand Down Expand Up @@ -155,8 +155,8 @@ libthrift-0.9.3.jar
log4j-1.2.17.jar
logging-interceptor-3.9.1.jar
lz4-java-1.5.0.jar
machinist_2.11-0.6.1.jar
macro-compat_2.11-1.1.1.jar
machinist_2.12-0.6.1.jar
macro-compat_2.12-1.1.1.jar
mesos-1.4.0-shaded-protobuf.jar
metrics-core-3.1.5.jar
metrics-graphite-3.1.5.jar
Expand Down Expand Up @@ -189,19 +189,19 @@ protobuf-java-2.5.0.jar
py4j-0.10.8.1.jar
pyrolite-4.13.jar
re2j-1.1.jar
scala-compiler-2.11.12.jar
scala-library-2.11.12.jar
scala-parser-combinators_2.11-1.1.0.jar
scala-reflect-2.11.12.jar
scala-xml_2.11-1.0.5.jar
shapeless_2.11-2.3.2.jar
scala-compiler-2.12.7.jar
scala-library-2.12.7.jar
scala-parser-combinators_2.12-1.1.0.jar
scala-reflect-2.12.7.jar
scala-xml_2.12-1.0.5.jar
shapeless_2.12-2.3.2.jar
slf4j-api-1.7.16.jar
slf4j-log4j12-1.7.16.jar
snakeyaml-1.18.jar
snappy-0.2.jar
snappy-java-1.1.7.1.jar
spire-macros_2.11-0.13.0.jar
spire_2.11-0.13.0.jar
spire-macros_2.12-0.13.0.jar
spire_2.12-0.13.0.jar
stax-api-1.0.1.jar
stax2-api-3.1.4.jar
stream-2.7.0.jar
Expand Down
4 changes: 2 additions & 2 deletions docs/_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ include:
# of Spark, Scala, and Mesos.
SPARK_VERSION: 3.0.0-SNAPSHOT
SPARK_VERSION_SHORT: 3.0.0
SCALA_BINARY_VERSION: "2.11"
SCALA_VERSION: "2.11.12"
SCALA_BINARY_VERSION: "2.12"
SCALA_VERSION: "2.12.7"
MESOS_VERSION: 1.0.0
SPARK_ISSUE_TRACKER_URL: https://issues.apache.org/jira/browse/SPARK
SPARK_GITHUB_URL: https://github.com/apache/spark
2 changes: 1 addition & 1 deletion docs/_plugins/copy_api_dirs.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@

# Copy over the unified ScalaDoc for all projects to api/scala.
# This directory will be copied over to _site when `jekyll` command is run.
source = "../target/scala-2.11/unidoc"
source = "../target/scala-2.12/unidoc"
dest = "api/scala"

puts "Making directory " + dest
Expand Down
18 changes: 9 additions & 9 deletions docs/building-spark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,9 @@ It's possible to build Spark submodules using the `mvn -pl` option.

For instance, you can build the Spark Streaming module using:

./build/mvn -pl :spark-streaming_2.11 clean install
./build/mvn -pl :spark-streaming_{{site.SCALA_BINARY_VERSION}} clean install

where `spark-streaming_2.11` is the `artifactId` as defined in `streaming/pom.xml` file.
where `spark-streaming_{{site.SCALA_BINARY_VERSION}}` is the `artifactId` as defined in `streaming/pom.xml` file.

## Continuous Compilation

Expand Down Expand Up @@ -230,25 +230,25 @@ Once installed, the `docker` service needs to be started, if not already running
On Linux, this can be done by `sudo service docker start`.

./build/mvn install -DskipTests
./build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.11
./build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_{{site.SCALA_BINARY_VERSION}}

or

./build/sbt docker-integration-tests/test

## Change Scala Version

To build Spark using another supported Scala version, please change the major Scala version using (e.g. 2.12):
To build Spark using another supported Scala version, please change the major Scala version using (e.g. 2.11):

./dev/change-scala-version.sh 2.12
./dev/change-scala-version.sh 2.11

For Maven, please enable the profile (e.g. 2.12):
For Maven, please enable the profile (e.g. 2.11):

./build/mvn -Pscala-2.12 compile
./build/mvn -Pscala-2.11 compile

For SBT, specify a complete scala version using (e.g. 2.12.6):
For SBT, specify a complete scala version using (e.g. 2.11.12):

./build/sbt -Dscala.version=2.12.6
./build/sbt -Dscala.version=2.11.12

Otherwise, the sbt-pom-reader plugin will use the `scala.version` specified in the spark-parent pom.

Expand Down
2 changes: 1 addition & 1 deletion docs/cloud-integration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ is set to the chosen version of Spark:
...
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>hadoop-cloud_2.11</artifactId>
<artifactId>hadoop-cloud_{{site.SCALA_BINARY_VERSION}}</artifactId>
<version>${spark.version}</version>
</dependency>
...
Expand Down
2 changes: 1 addition & 1 deletion docs/sparkr.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ specifying `--packages` with `spark-submit` or `sparkR` commands, or if initiali

<div data-lang="r" markdown="1">
{% highlight r %}
sparkR.session(sparkPackages = "com.databricks:spark-avro_2.11:3.0.0")
sparkR.session(sparkPackages = "org.apache.spark:spark-avro_{{site.SCALA_BINARY_VERSION}}:{{site.SPARK_VERSION}}")
{% endhighlight %}
</div>

Expand Down
4 changes: 2 additions & 2 deletions examples/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>spark-examples_2.11</artifactId>
<artifactId>spark-examples_2.12</artifactId>
<packaging>jar</packaging>
<name>Spark Project Examples</name>
<url>http://spark.apache.org/</url>
Expand Down
Loading