[SPARK-1776] Have Spark's SBT build read dependencies from Maven. #772

ScrapCodes · 2014-05-14T13:32:25Z

Patch introduces the new way of working also retaining the existing ways of doing things.

For example build instruction for yarn in maven is
mvn -Pyarn -PHadoop2.2 clean package -DskipTests
in sbt it can become
MAVEN_PROFILES="yarn, hadoop-2.2" sbt/sbt clean assembly
Also supports
sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 clean assembly

AmplabJenkins · 2014-05-14T13:32:58Z

Build triggered.

AmplabJenkins · 2014-05-14T13:33:03Z

Build started.

AmplabJenkins · 2014-05-14T13:42:57Z

Merged build triggered.

AmplabJenkins · 2014-05-14T13:43:03Z

Merged build started.

AmplabJenkins · 2014-05-14T15:03:47Z

Build finished.

AmplabJenkins · 2014-05-14T15:03:47Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14980/

AmplabJenkins · 2014-05-14T15:13:47Z

Merged build finished.

AmplabJenkins · 2014-05-14T15:13:47Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14981/

pwendell · 2014-05-15T07:03:09Z

Hey @ScrapCodes - this looks like a good start. In terms of how to pass options I think the nicest behavior would be to just make it identical to maven:

sbt/sbt -Pyarn -Dspark.hadoop.version=1.2.3 ~compile

Since we control the sbt/sbt script we could do this (I think) by just capturing the arguments and then putting them in an environment variable or passing them somehow so they can be read in the sbt build.

It would also be good, for backwards compatibility, to do some automatic conversions:

if SPARK_YARN is set ==> enable 'yarn' profile
if SPARK_GANGLIA_LGPL is set => enable 'spark-ganglia-lgpl' profile
if SPARK_HIVE is set => enable the 'hive' profile
if SPARK_HADOOP_VERSION is set ==> spark.hadoop.version property

AmplabJenkins · 2014-05-19T11:12:57Z

Merged build triggered.

AmplabJenkins · 2014-05-19T11:13:05Z

Merged build started.

AmplabJenkins · 2014-05-19T12:43:53Z

Merged build finished.

AmplabJenkins · 2014-05-19T12:43:54Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15080/

ScrapCodes · 2014-05-19T12:48:06Z

@pwendell Haven't worked on backward compatibility yet, other things you pointed out should work fine. Also the bug with tests dependencies in streaming got fixed. Now sbt/sbt ~test:compile works !.

AmplabJenkins · 2014-05-20T07:12:57Z

Build triggered.

AmplabJenkins · 2014-05-20T07:13:06Z

Build started.

AmplabJenkins · 2014-05-20T07:17:57Z

Build triggered.

AmplabJenkins · 2014-05-20T07:18:06Z

Build started.

AmplabJenkins · 2014-05-20T07:22:58Z

Build triggered.

AmplabJenkins · 2014-05-20T07:23:06Z

Build started.

AmplabJenkins · 2014-05-20T07:27:57Z

Build triggered.

AmplabJenkins · 2014-05-20T07:28:06Z

Build started.

AmplabJenkins · 2014-05-20T07:40:14Z

Build finished.

AmplabJenkins · 2014-05-20T07:40:14Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15099/

AmplabJenkins · 2014-05-20T07:42:57Z

Build triggered.

AmplabJenkins · 2014-05-20T07:43:06Z

Build started.

AmplabJenkins · 2014-05-20T07:44:37Z

Build finished.

AmplabJenkins · 2014-05-20T07:44:37Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15100/

AmplabJenkins · 2014-05-20T07:48:39Z

Build finished.

AmplabJenkins · 2014-05-20T07:48:39Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15098/

AmplabJenkins · 2014-07-09T22:51:14Z

Merged build triggered.

AmplabJenkins · 2014-07-09T22:51:21Z

Merged build started.

AmplabJenkins · 2014-07-10T00:31:12Z

Merged build finished.

AmplabJenkins · 2014-07-10T00:31:12Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16485/

pwendell · 2014-07-10T05:34:57Z

@ScrapCodes could you try to figure out why the test reports aren't published? There are a few other patches blocking on this, so we should really merge it.

pwendell · 2014-07-10T05:37:27Z

@ScrapCodes it might be good to look through the source code of the XML report plug-in and see how it works. You could also inspect the different configs that are set during tests. You can also see if using a different plug-in will work:

https://github.com/bseibel/sbt-simple-junit-xml-reporter-plugin

AmplabJenkins · 2014-07-10T08:11:14Z

Merged build triggered.

AmplabJenkins · 2014-07-10T08:11:23Z

Merged build started.

SparkQA · 2014-07-10T08:12:33Z

QA tests have started for PR 772. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16494/consoleFull

AmplabJenkins · 2014-07-10T10:12:38Z

Merged build finished.

AmplabJenkins · 2014-07-10T10:12:38Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16494/

ScrapCodes · 2014-07-10T10:13:24Z

Jenkins, retest this please.

AmplabJenkins · 2014-07-10T10:16:24Z

Merged build started.

AmplabJenkins · 2014-07-10T10:16:25Z

Merged build triggered.

SparkQA · 2014-07-10T10:17:26Z

QA tests have started for PR 772. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16497/consoleFull

SparkQA · 2014-07-10T11:54:54Z

QA results for PR 772:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16497/consoleFull

AmplabJenkins · 2014-07-10T11:54:59Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-10T11:54:59Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16497/

ScrapCodes · 2014-07-10T12:14:13Z

I have updated the sbt version and that fixed the problem.

pwendell · 2014-07-10T18:03:25Z

Great! I'm going to merge this (yay!!).

Patch introduces the new way of working also retaining the existing ways of doing things. For example build instruction for yarn in maven is `mvn -Pyarn -PHadoop2.2 clean package -DskipTests` in sbt it can become `MAVEN_PROFILES="yarn, hadoop-2.2" sbt/sbt clean assembly` Also supports `sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 clean assembly` Author: Prashant Sharma <[email protected]> Author: Patrick Wendell <[email protected]> Closes apache#772 from ScrapCodes/sbt-maven and squashes the following commits: a8ac951 [Prashant Sharma] Updated sbt version. 62b09bb [Prashant Sharma] Improvements. fa6221d [Prashant Sharma] Excluding sql from mima 4b8875e [Prashant Sharma] Sbt assembly no longer builds tools by default. 72651ca [Prashant Sharma] Addresses code reivew comments. acab73d [Prashant Sharma] Revert "Small fix to run-examples script." ac4312c [Prashant Sharma] Revert "minor fix" 6af91ac [Prashant Sharma] Ported oldDeps back. + fixes issues with prev commit. 65cf06c [Prashant Sharma] Servelet API jars mess up with the other servlet jars on the class path. 446768e [Prashant Sharma] minor fix 89b9777 [Prashant Sharma] Merge conflicts d0a02f2 [Prashant Sharma] Bumped up pom versions, Since the build now depends on pom it is better updated there. + general cleanups. dccc8ac [Prashant Sharma] updated mima to check against 1.0 a49c61b [Prashant Sharma] Fix for tools jar a2f5ae1 [Prashant Sharma] Fixes a bug in dependencies. cf88758 [Prashant Sharma] cleanup 9439ea3 [Prashant Sharma] Small fix to run-examples script. 96cea1f [Prashant Sharma] SPARK-1776 Have Spark's SBT build read dependencies from Maven. 36efa62 [Patrick Wendell] Set project name in pom files and added eclipse/intellij plugins. 4973dbd [Patrick Wendell] Example build using pom reader.

Updated sbt version.

a8ac951

asfgit closed this in 628932b Jul 10, 2014

witgo mentioned this pull request Jul 16, 2014

SPARK-2482: Resolve sbt warnings during build #1330

Closed

witgo mentioned this pull request Aug 3, 2014

[SPARK-2815]: Compilation failed upon the hadoop version 2.0.0-cdh4.5.0 #1754

Closed

ScrapCodes deleted the sbt-maven branch June 3, 2015 06:10

[SPARK-1776] Have Spark's SBT build read dependencies from Maven. #772

[SPARK-1776] Have Spark's SBT build read dependencies from Maven. #772

Uh oh!

Conversation

ScrapCodes commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

pwendell commented May 15, 2014

Uh oh!

AmplabJenkins commented May 19, 2014

Uh oh!

AmplabJenkins commented May 19, 2014

Uh oh!

AmplabJenkins commented May 19, 2014

Uh oh!

AmplabJenkins commented May 19, 2014

Uh oh!

ScrapCodes commented May 19, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented May 20, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 9, 2014

Uh oh!

AmplabJenkins commented Jul 10, 2014

Uh oh!

AmplabJenkins commented Jul 10, 2014

Uh oh!

pwendell commented Jul 10, 2014

Uh oh!

pwendell commented Jul 10, 2014

Uh oh!

AmplabJenkins commented Jul 10, 2014

Uh oh!