Skip to content

[SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work #969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
witgo wants to merge 19 commits into from
Closed

[SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work #969

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
8362489
yarn.ClientBase spark.yarn.dist.* do not work
witgo Jun 4, 2014
ce609fc
Merge branch 'master' into yarn_ClientBase
witgo Jun 9, 2014
0961151
merge master
witgo Jun 11, 2014
20e667c
Merge branch 'master' into yarn_ClientBase
witgo Jun 13, 2014
8bc2f4b
review commit
witgo Jun 13, 2014
9cdff16
review commit
witgo Jun 13, 2014
55d72fc
Merge branch 'master' of https://github.com/apache/spark into yarn_Cl…
witgo Jun 13, 2014
35d6fa0
move to ClientArguments
witgo Jun 13, 2014
41bce59
review commit
witgo Jun 13, 2014
871f1db
add spark.yarn.dist.* documentation
witgo Jun 13, 2014
1048549
remove Utils.resolveURIs
witgo Jun 15, 2014
8d7b82f
Merge branch 'master' of https://github.com/apache/spark into yarn_Cl…
witgo Jun 15, 2014
2f48789
Merge branch 'master' of https://github.com/apache/spark into yarn_Cl…
witgo Jun 16, 2014
c8b4554
review commit
witgo Jun 16, 2014
b6a9aa1
merge master
witgo Jun 17, 2014
e3c1107
update docs
witgo Jun 17, 2014
5261b6c
fix sys.props.get("SPARK_YARN_DIST_FILES")
witgo Jun 17, 2014
3bdbc52
Merge branch 'master' of https://github.com/apache/spark into yarn_Cl…
witgo Jun 17, 2014
8117765
review commit
witgo Jun 17, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,10 +220,21 @@ trait ClientBase extends Logging {
}
}

def getArg(arg: String, envVar: String, sysProp: String): String = {
if (arg != null && !arg.isEmpty) {
arg
} else if (System.getenv(envVar) != null && !System.getenv(envVar).isEmpty) {
System.getenv(envVar)
} else {
sparkConf.getOption(sysProp).orNull
}
}
var cachedSecondaryJarLinks = ListBuffer.empty[String]
val fileLists = List( (args.addJars, LocalResourceType.FILE, true),
(args.files, LocalResourceType.FILE, false),
(args.archives, LocalResourceType.ARCHIVE, false) )
val fileLists = List((args.addJars, LocalResourceType.FILE, true),
(getArg(args.files, "SPARK_YARN_DIST_FILES", "spark.yarn.dist.files"),
LocalResourceType.FILE, false),
(getArg(args.archives, "SPARK_YARN_DIST_ARCHIVES", "spark.yarn.dist.archives"),
Copy link
Contributor

@vanzin vanzin Jun 4, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think env variables and conf entries should be handled here like this.

YarnClientSchedulerBackend already deals with the env variable and command line option for client mode. It seems that SparkSubmit might be missing code to handle the env variable for cluster mode, though. Probably better to fix it there, and leave this code to deal only with the command line args (which are already correctly parsed).

Copy link
Contributor Author

@witgo witgo Jun 12, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vanzin This is a yarn cluster problems. Don't use YarnClientSchedulerBackend.

Copy link
Contributor

@vanzin vanzin Jun 12, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you see the other PR I mentioned (see #969 (comment))?

There's a bug in the Yarn code where configuration is not propagated correctly in cluster mode, which I fixed there. So you don't need this here because with my fix, these options should be correctly propagated without any extra code.

Copy link
Contributor Author

@witgo witgo Jun 13, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, Weekend, I will test your branch, Please merge the master changes.

LocalResourceType.ARCHIVE, false))
fileLists.foreach { case (flist, resType, addToClasspath) =>
if (flist != null && !flist.isEmpty()) {
flist.split(',').foreach { case file: String =>
Expand Down