Skip to content

Assorted clean-up for Spark-on-YARN. #488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pwendell wants to merge 3 commits into from
Closed

Assorted clean-up for Spark-on-YARN. #488

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
17929cc
Assorted clean-up for Spark-on-YARN.
pwendell Apr 22, 2014
18d09c1
Review comments from Andrew
pwendell Apr 22, 2014
fe95f13
Changes based on Andrew's feeback
pwendell Apr 23, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions conf/spark-env.sh.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

# Options read when launching programs locally with
# ./bin/run-example or ./bin/spark-submit
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
Copy link
Contributor

@andrewor14 andrewor14 Apr 22, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be under "Options read in YARN client mode"?

Copy link
Contributor Author

@pwendell pwendell Apr 22, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added it there as well, but I'm going to leave it here because it's relevant also to yarn-cluster.

# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public dns name of the driver program
# - SPARK_CLASSPATH, default classpath entries to append
Expand All @@ -17,6 +18,7 @@
# - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos

# Options read in YARN client mode
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_EXECUTOR_INSTANCES, Number of workers to start (Default: 2)
# - SPARK_EXECUTOR_CORES, Number of cores for the workers (Default: 1).
# - SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G)
Expand Down
11 changes: 11 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,17 @@ private[spark] class SparkSubmitArguments(args: Array[String]) {
if (args.length == 0) printUsageAndExit(-1)
if (primaryResource == null) SparkSubmit.printErrorAndExit("Must specify a primary resource")
if (mainClass == null) SparkSubmit.printErrorAndExit("Must specify a main class with --class")
val testing = sys.env.contains("SPARK_TESTING")
if (master.startsWith("yarn")) {
val hasHadoopEnv = testing ||
sys.env.contains("HADOOP_CONF_DIR") ||
sys.env.contains("YARN_CONF_DIR")

if (!hasHadoopEnv) {
Copy link
Contributor

@andrewor14 andrewor14 Apr 23, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super small nit: I meant something more like if (!hasHadoopEnv && !testing) here, since the variable hasHadoopEnv should really not depend on whether we're testing Spark or not.

Copy link
Contributor Author

@pwendell pwendell Apr 23, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah that makes way more sense. Thanks

throw new Exception(s"When running with master '$master' " +
"either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.")
}
}
}

override def toString = {
Expand Down
9 changes: 2 additions & 7 deletions docs/hadoop-third-party-distributions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,5 @@ The location of these configuration files varies across CDH and HDP versions, bu
a common location is inside of `/etc/hadoop/conf`. Some tools, such as Cloudera Manager, create
configurations on-the-fly, but offer a mechanisms to download copies of them.

There are a few ways to make these files visible to Spark:

* You can copy these files into `$SPARK_HOME/conf` and they will be included in Spark's
classpath automatically.
* If you are running Spark on the same nodes as Hadoop _and_ your distribution includes both
`hdfs-site.xml` and `core-site.xml` in the same directory, you can set `HADOOP_CONF_DIR`
in `$SPARK_HOME/spark-env.sh` to that directory.
To make these files visible to Spark, set `HADOOP_CONF_DIR` in `$SPARK_HOME/spark-env.sh`
to a location containing the configuration files.
8 changes: 5 additions & 3 deletions yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -375,9 +375,11 @@ object ClientBase {
val classpathEntries = Option(conf.getStrings(
YarnConfiguration.YARN_APPLICATION_CLASSPATH)).getOrElse(
getDefaultYarnApplicationClasspath())
for (c <- classpathEntries) {
YarnSparkHadoopUtil.addToEnvironment(env, Environment.CLASSPATH.name, c.trim,
File.pathSeparator)
if (classpathEntries != null) {
for (c <- classpathEntries) {
YarnSparkHadoopUtil.addToEnvironment(env, Environment.CLASSPATH.name, c.trim,
File.pathSeparator)
}
}

val mrClasspathEntries = Option(conf.getStrings(
Expand Down