Skip to content

Fix SPARK-1609: Executor fails to start when Command.extraJavaOptions contains multiple Java options #547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
witgo wants to merge 9 commits into from
Closed

Fix SPARK-1609: Executor fails to start when Command.extraJavaOptions contains multiple Java options #547

Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8a265b7
Fix SPARK-1609: Executor fails to start when use spark-submit
witgo Apr 24, 2014
86fc4bb
bugfix
witgo Apr 25, 2014
f7c0ab7
bugfix
witgo Apr 25, 2014
1185605
fix extraJavaOptions split
witgo Apr 25, 2014
bcf36cb
Merge branch 'master' of https://github.com/apache/spark into SPARK-1609
witgo Apr 25, 2014
8f90b22
Merge branch 'master' of https://github.com/apache/spark into SPARK-1609
witgo Apr 25, 2014
0640852
review commit
witgo Apr 25, 2014
91da0bb
Merge branch 'master' of https://github.com/apache/spark into SPARK-1609
witgo Apr 26, 2014
deb6a4c
review commit
witgo Apr 26, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,13 @@ object CommandUtils extends Logging {
def buildJavaOpts(command: Command, memory: Int, sparkHome: String): Seq[String] = {
val memoryOpts = Seq(s"-Xms${memory}M", s"-Xmx${memory}M")
// Note, this will coalesce multiple options into a single command component
val extraOpts = command.extraJavaOptions.toSeq
val extraOpts = command.extraJavaOptions match {
Copy link
Contributor

@pwendell pwendell Apr 26, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Up to you, but could be written a bit more consicely

val extraOpts = command.extraJavaOptions.map(Utils.splitCommandString).getOrElse(Seq())

Copy link
Contributor Author

@witgo witgo Apr 26, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, val extraOpts = command.extraJavaOptions.map(Utils.splitCommandString).getOrElse(Seq()) is better

case Some(opts) =>
opts.split("\\s+").toSeq
Copy link
Contributor

@mateiz mateiz Apr 25, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use Utils.splitCommandString instead of this, in case people pass in stuff with quotes.

case _ =>
Seq()
}

val libraryOpts =
if (command.libraryPathEntries.size > 0) {
val joined = command.libraryPathEntries.mkString(File.pathSeparator)
Expand All @@ -62,10 +68,10 @@ object CommandUtils extends Logging {
val classPath = Utils.executeAndGetOutput(
Seq(sparkHome + "/bin/compute-classpath" + ext),
extraEnvironment=command.environment)
val userClassPath = command.classPathEntries.mkString(File.pathSeparator)
val classPathWithUser = classPath + File.pathSeparator + userClassPath
val userClassPath = command.classPathEntries ++ Seq(classPath)

Seq("-cp", classPathWithUser) ++ libraryOpts ++ extraOpts ++ memoryOpts
Seq("-cp", userClassPath.filterNot(_.isEmpty).mkString(File.pathSeparator)) ++
libraryOpts ++ extraOpts ++ memoryOpts
}

/** Spawn a thread that will redirect a given stream to a file */
Expand Down