Skip to content

[SPARK-3437][BUILD] Support crossbuilding in maven. With new scala-install-plugin. #2357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ScrapCodes wants to merge 2 commits into from
Closed

[SPARK-3437][BUILD] Support crossbuilding in maven. With new scala-install-plugin. #2357

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4d2d30c
Supported new scala install plugin. Which can let us cross build for …
ScrapCodes Sep 11, 2014
609dd98
Changed to newly updated with cross build support branch.
ScrapCodes Sep 11, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-assembly_2.10</artifactId>
<artifactId>spark-assembly</artifactId>
<name>Spark Project Assembly</name>
<url>http://spark.apache.org/</url>
<packaging>pom</packaging>
Expand Down
2 changes: 1 addition & 1 deletion bagel/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-bagel_2.10</artifactId>
<artifactId>spark-bagel</artifactId>
<properties>
<sbt.project.name>bagel</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<artifactId>spark-core</artifactId>
<properties>
<sbt.project.name>core</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion examples/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-examples_2.10</artifactId>
<artifactId>spark-examples</artifactId>
<properties>
<sbt.project.name>examples</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion external/flume-sink/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-flume-sink_2.10</artifactId>
<artifactId>spark-streaming-flume-sink</artifactId>
<properties>
<sbt.project.name>streaming-flume-sink</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion external/flume/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-flume_2.10</artifactId>
<artifactId>spark-streaming-flume</artifactId>
<properties>
<sbt.project.name>streaming-flume</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion external/kafka/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.10</artifactId>
<artifactId>spark-streaming-kafka</artifactId>
<properties>
<sbt.project.name>streaming-kafka</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion external/mqtt/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-mqtt_2.10</artifactId>
<artifactId>spark-streaming-mqtt</artifactId>
<properties>
<sbt.project.name>streaming-mqtt</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion external/twitter/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-twitter_2.10</artifactId>
<artifactId>spark-streaming-twitter</artifactId>
<properties>
<sbt.project.name>streaming-twitter</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion external/zeromq/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-zeromq_2.10</artifactId>
<artifactId>spark-streaming-zeromq</artifactId>
<properties>
<sbt.project.name>streaming-zeromq</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion extras/java8-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>java8-tests_2.10</artifactId>
<artifactId>java8-tests</artifactId>
<packaging>pom</packaging>
<name>Spark Project Java8 Tests POM</name>

Expand Down
2 changes: 1 addition & 1 deletion extras/kinesis-asl/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

<!-- Kinesis integration is not included by default due to ASL-licensed code. -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kinesis-asl_2.10</artifactId>
<artifactId>spark-streaming-kinesis-asl</artifactId>
<packaging>jar</packaging>
<name>Spark Kinesis Integration</name>

Expand Down
2 changes: 1 addition & 1 deletion extras/spark-ganglia-lgpl/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

<!-- Ganglia integration is not included by default due to LGPL-licensed code -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-ganglia-lgpl_2.10</artifactId>
<artifactId>spark-ganglia-lgpl</artifactId>
<packaging>jar</packaging>
<name>Spark Ganglia Integration</name>

Expand Down
2 changes: 1 addition & 1 deletion graphx/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-graphx_2.10</artifactId>
<artifactId>spark-graphx</artifactId>
<properties>
<sbt.project.name>graphx</sbt.project.name>
</properties>
Expand Down
2 changes: 1 addition & 1 deletion mllib/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<artifactId>spark-mllib</artifactId>
<properties>
<sbt.project.name>mllib</sbt.project.name>
</properties>
Expand Down
39 changes: 37 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,18 +89,18 @@
<module>core</module>
<module>bagel</module>
<module>graphx</module>
<module>streaming</module>
<module>mllib</module>
<module>tools</module>
<module>streaming</module>
<module>sql/catalyst</module>
<module>sql/core</module>
<module>sql/hive</module>
<module>repl</module>
<module>assembly</module>
<module>external/twitter</module>
<module>external/kafka</module>
<module>external/flume</module>
<module>external/flume-sink</module>
<module>external/flume</module>
<module>external/zeromq</module>
<module>external/mqtt</module>
<module>examples</module>
Expand Down Expand Up @@ -774,9 +774,20 @@
</dependencyManagement>

<build>
<finalName>
${project.artifactId}_${scala.binary.version}-${project.version}
</finalName>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.3.1</version>
Expand Down Expand Up @@ -961,6 +972,30 @@
</pluginManagement>

<plugins>
<plugin>
<groupId>org.apache.spark</groupId>
<artifactId>scala-install-plugin</artifactId>
<version>0.1-SNAPSHOT</version>
<executions>
<execution>
<id>install</id>
<goals>
<goal>install</goal>
</goals>
<phase>install</phase>
<configuration>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
</configuration>
</execution>
<execution>
<id>load</id>
<goals>
<goal>init-scala</goal>
</goals>
<phase>initialize</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion project/project/SparkPluginBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import sbt.Keys._
object SparkPluginDef extends Build {
lazy val root = Project("plugins", file(".")) dependsOn(sparkStyle, sbtPomReader)
lazy val sparkStyle = Project("spark-style", file("spark-style"), settings = styleSettings)
lazy val sbtPomReader = uri("https://github.com/ScrapCodes/sbt-pom-reader.git")
lazy val sbtPomReader = uri("https://github.com/ScrapCodes/sbt-pom-reader.git#wip-cross-build")

// There is actually no need to publish this artifact.
def styleSettings = Defaults.defaultSettings ++ Seq (
Expand Down
2 changes: 1 addition & 1 deletion repl/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-repl_2.10</artifactId>
<artifactId>spark-repl</artifactId>
<packaging>jar</packaging>
<name>Spark Project REPL</name>
<url>http://spark.apache.org/</url>
Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_2.10</artifactId>
<artifactId>spark-catalyst</artifactId>
<packaging>jar</packaging>
<name>Spark Project Catalyst</name>
<url>http://spark.apache.org/</url>
Expand Down
2 changes: 1 addition & 1 deletion sql/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<artifactId>spark-sql</artifactId>
<packaging>jar</packaging>
<name>Spark Project SQL</name>
<url>http://spark.apache.org/</url>
Expand Down
2 changes: 1 addition & 1 deletion sql/hive-thriftserver/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-hive-thriftserver_2.10</artifactId>
<artifactId>spark-hive-thriftserver</artifactId>
<packaging>jar</packaging>
<name>Spark Project Hive Thrift Server</name>
<url>http://spark.apache.org/</url>
Expand Down
2 changes: 1 addition & 1 deletion sql/hive/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<artifactId>spark-hive</artifactId>
<packaging>jar</packaging>
<name>Spark Project Hive</name>
<url>http://spark.apache.org/</url>
Expand Down
4 changes: 2 additions & 2 deletions streaming/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<artifactId>spark-streaming</artifactId>
<properties>
<sbt.project.name>streaming</sbt.project.name>
</properties>
Expand Down Expand Up @@ -77,7 +77,7 @@
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
</plugin>

<!--
This plugin forces the generation of jar containing streaming test classes,
so that the tests classes of external modules can use them. The two execution profiles
Expand Down
2 changes: 1 addition & 1 deletion tools/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-tools_2.10</artifactId>
<artifactId>spark-tools</artifactId>
<properties>
<sbt.project.name>tools</sbt.project.name>
</properties>
Expand Down
4 changes: 2 additions & 2 deletions yarn/alpha/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>yarn-parent_2.10</artifactId>
<artifactId>yarn-parent</artifactId>
<version>1.2.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
Expand All @@ -28,7 +28,7 @@
</properties>

<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn-alpha_2.10</artifactId>
<artifactId>spark-yarn-alpha</artifactId>
<packaging>jar</packaging>
<name>Spark Project YARN Alpha API</name>

Expand Down
2 changes: 1 addition & 1 deletion yarn/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>yarn-parent_2.10</artifactId>
<artifactId>yarn-parent</artifactId>
<packaging>pom</packaging>
<name>Spark Project YARN Parent POM</name>
<properties>
Expand Down
4 changes: 2 additions & 2 deletions yarn/stable/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>yarn-parent_2.10</artifactId>
<artifactId>yarn-parent</artifactId>
<version>1.2.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
Expand All @@ -28,7 +28,7 @@
</properties>

<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn_2.10</artifactId>
<artifactId>spark-yarn</artifactId>
<packaging>jar</packaging>
<name>Spark Project YARN Stable API</name>

Expand Down