Skip to content

Commit 40dbc0b

Browse files
author
Neelesh Srinivas Salian
committed
Changed dfs to HDFS, deploy-mode in backticks and updated the master yarn line
1 parent 9cbc072 commit 40dbc0b

1 file changed

Lines changed: 3 additions & 3 deletions

File tree

docs/running-on-yarn.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,16 +10,16 @@ was added to Spark in version 0.6.0, and improved in subsequent releases.
1010
# Launching Spark on YARN
1111

1212
Ensure that `HADOOP_CONF_DIR` or `YARN_CONF_DIR` points to the directory which contains the (client side) configuration files for the Hadoop cluster.
13-
These configs are used to write to the dfs and connect to the YARN ResourceManager. The
13+
These configs are used to write to HDFS and connect to the YARN ResourceManager. The
1414
configuration contained in this directory will be distributed to the YARN cluster so that all
1515
containers used by the application use the same configuration. If the configuration references
1616
Java system properties or environment variables not managed by YARN, they should also be set in the
1717
Spark application's configuration (driver, executors, and the AM when running in client mode).
1818

1919
There are two deploy modes that can be used to launch Spark applications on YARN. In yarn-cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In yarn-client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.
20-
(Default: --deploy-mode client)
20+
(Default: `--deploy-mode client`)
2121

22-
Unlike in Spark standalone and Mesos mode, in which the master's address is specified in the "master" parameter, in YARN mode the ResourceManager's address is picked up from the Hadoop configuration. Thus, the master parameter is yarn.
22+
Unlike in Spark standalone and Mesos mode, in which the master's address is specified in the "master" parameter, in YARN mode the ResourceManager's address is picked up from the Hadoop configuration. Thus, the master parameter is yarn. For a specific yarn deployment, use --deploy-mode to specify yarn-cluster or yarn-client.
2323

2424
To launch a Spark application in yarn-cluster mode:
2525

0 commit comments

Comments
 (0)