Skip to content

[SPARK-3437][BUILD] Support crossbuilding in maven. #2318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ScrapCodes wants to merge 1 commit into from
Closed

[SPARK-3437][BUILD] Support crossbuilding in maven. #2318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-assembly_2.10</artifactId>
<artifactId>spark-assembly</artifactId>
Copy link
Member

@srowen srowen Sep 8, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It goes without saying, but this introduces a sort of 'incompatibility' at the Maven level. Downstream projects have to change artifact and version when selecting between this and a future version with different artifact names. This doesn't automatically harmonize the Spark build with the version of Scala you're using, and Maven can't resolve dependency conflicts across differently named artifacts. These are kind of inevitable.

If an end goal is to publicly support Scala 2.10 and 2.11 simultaneously, then we still have to have several published artifacts at once right? this is what classifier could be used for. But I suppose it's worth asking if that's the intent? or is this going to may be happen all at once for Spark 2.x?

Copy link
Contributor

@pwendell pwendell Sep 8, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sean I think this approach actually modifies the published pom's to look different than the poms which are our own build. I.e. it appends the 2.11 or 2.10 suffix to the artifact in the published poms. Other than being horrendously verbose... this does seem to be a nicer approach IMO.

Since donstream projects only consume spark via the published poms, it seems okay to me. What is the specific case you're concerned about?

Copy link
Member

@srowen srowen Sep 8, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see now. That makes sense then, and leaves open the possibility of publishing parallel builds for Scala 2.10 / 2.11 under the current naming convention. Although maybe you could argue for using classifier instead, there's no point changing conventions if they're not already being changed.

Copy link
Member Author

@ScrapCodes ScrapCodes Sep 9, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just checked the published pom had, just this inside it.

<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.10</artifactId>
  <version>1.2.0-SNAPSHOT</version>
  <description>POM was created from install:install-file</description>
</project>

No dependency information, I am not sure if we can live without that.

<name>Spark Project Assembly</name>
<url>http://spark.apache.org/</url>
<packaging>pom</packaging>
Expand All @@ -51,37 +51,37 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<artifactId>spark-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-bagel_${scala.binary.version}</artifactId>
<artifactId>spark-bagel</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.binary.version}</artifactId>
<artifactId>spark-mllib</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-repl_${scala.binary.version}</artifactId>
<artifactId>spark-repl</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.binary.version}</artifactId>
<artifactId>spark-streaming</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-graphx_${scala.binary.version}</artifactId>
<artifactId>spark-graphx</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<artifactId>spark-sql</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand Down Expand Up @@ -158,7 +158,7 @@
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn-alpha_${scala.binary.version}</artifactId>
<artifactId>spark-yarn-alpha</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand All @@ -168,7 +168,7 @@
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn_${scala.binary.version}</artifactId>
<artifactId>spark-yarn</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand All @@ -178,12 +178,12 @@
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<artifactId>spark-hive</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive-thriftserver_${scala.binary.version}</artifactId>
<artifactId>spark-hive-thriftserver</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand All @@ -193,7 +193,7 @@
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-ganglia-lgpl_${scala.binary.version}</artifactId>
<artifactId>spark-ganglia-lgpl</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand Down
51 changes: 49 additions & 2 deletions bagel/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-bagel_2.10</artifactId>
<artifactId>spark-bagel</artifactId>
<properties>
<sbt.project.name>bagel</sbt.project.name>
</properties>
Expand All @@ -37,7 +37,7 @@
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<artifactId>spark-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
Expand All @@ -59,6 +59,53 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
Copy link
Member

@srowen srowen Sep 8, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should specify this version once in parent pom in pluginManagement

<executions>
<execution>
<id>1</id>
<goals>
<goal>install-file</goal>
</goals>
<phase>install</phase>
<configuration>
<file>${project.build.directory}/${project.build.finalName}.jar</file>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
</configuration>
</execution>
<execution>
<id>2</id>
<goals>
<goal>install-file</goal>
</goals>
<phase>install</phase>
<configuration>
<generatePom>false</generatePom>
<pomFile>pom.xml</pomFile>
<file>${project.build.directory}/${project.build.finalName}-sources.jar</file>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
<classifier>sources</classifier>
</configuration>
</execution>
<execution>
<id>3</id>
<goals>
<goal>install-file</goal>
</goals>
<phase>install</phase>
<configuration>
<generatePom>false</generatePom>
<pomFile>pom.xml</pomFile>
Copy link
Member

@srowen srowen Sep 8, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact it looks like all of this config can be common in the parent since it looks like it's written generically in terms of properties?

<file>${project.build.directory}/${project.build.finalName}-javadoc.jar</file>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
<classifier>javadoc</classifier>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
Expand Down
50 changes: 49 additions & 1 deletion core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<artifactId>spark-core</artifactId>
<properties>
<sbt.project.name>core</sbt.project.name>
</properties>
Expand Down Expand Up @@ -351,6 +351,54 @@
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
<executions>
<execution>
<id>1</id>
<goals>
<goal>install-file</goal>
</goals>
<phase>install</phase>
<configuration>
<file>${project.build.directory}/${project.build.finalName}.jar</file>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
</configuration>
</execution>
<execution>
<id>2</id>
<goals>
<goal>install-file</goal>
</goals>
<phase>install</phase>
<configuration>
<generatePom>false</generatePom>
<pomFile>pom.xml</pomFile>
<file>${project.build.directory}/${project.build.finalName}-sources.jar</file>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
<classifier>sources</classifier>
</configuration>
</execution>
<execution>
<id>3</id>
<goals>
<goal>install-file</goal>
</goals>
<phase>install</phase>
<configuration>
<generatePom>false</generatePom>
<pomFile>pom.xml</pomFile>
<file>${project.build.directory}/${project.build.finalName}-javadoc.jar</file>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
<classifier>javadoc</classifier>
</configuration>
</execution>
</executions>
</plugin>

</plugins>

<resources>
Expand Down
2 changes: 1 addition & 1 deletion dev/audit-release/maven_app_core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
<dependencies>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<artifactId>spark-core</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
Expand Down
73 changes: 60 additions & 13 deletions examples/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-examples_2.10</artifactId>
<artifactId>spark-examples</artifactId>
<properties>
<sbt.project.name>examples</sbt.project.name>
</properties>
Expand All @@ -40,7 +40,7 @@
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kinesis-asl_${scala.binary.version}</artifactId>
<artifactId>spark-streaming-kinesis-asl</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand All @@ -56,63 +56,63 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<artifactId>spark-core</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.binary.version}</artifactId>
<artifactId>spark-streaming</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.binary.version}</artifactId>
<artifactId>spark-mllib</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-bagel_${scala.binary.version}</artifactId>
<artifactId>spark-bagel</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<artifactId>spark-hive</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-graphx_${scala.binary.version}</artifactId>
<artifactId>spark-graphx</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-twitter_${scala.binary.version}</artifactId>
<artifactId>spark-streaming-twitter</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_${scala.binary.version}</artifactId>
<artifactId>spark-streaming-kafka</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-flume_${scala.binary.version}</artifactId>
<artifactId>spark-streaming-flume</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-zeromq_${scala.binary.version}</artifactId>
<artifactId>spark-streaming-zeromq</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-mqtt_${scala.binary.version}</artifactId>
<artifactId>spark-streaming-mqtt</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
Expand Down Expand Up @@ -203,6 +203,53 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
<executions>
<execution>
<id>1</id>
<goals>
<goal>install-file</goal>
</goals>
<phase>install</phase>
<configuration>
<file>${project.build.directory}/${project.build.finalName}.jar</file>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
</configuration>
</execution>
<execution>
<id>2</id>
<goals>
<goal>install-file</goal>
</goals>
<phase>install</phase>
<configuration>
<generatePom>false</generatePom>
<pomFile>pom.xml</pomFile>
<file>${project.build.directory}/${project.build.finalName}-sources.jar</file>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
<classifier>sources</classifier>
</configuration>
</execution>
<execution>
<id>3</id>
<goals>
<goal>install-file</goal>
</goals>
<phase>install</phase>
<configuration>
<generatePom>false</generatePom>
<pomFile>pom.xml</pomFile>
<file>${project.build.directory}/${project.build.finalName}-javadoc.jar</file>
<artifactId>${project.artifactId}_${scala.binary.version}</artifactId>
<classifier>javadoc</classifier>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
Expand Down
Loading