Skip to content

[SPARK-11653][Deploy] Allow spark-daemon.sh to run in the foreground #15338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mikejihbe wants to merge 3 commits into from
Closed

[SPARK-11653][Deploy] Allow spark-daemon.sh to run in the foreground #15338

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cb89755
[SPARK-11653][Deploy] Add a SPARK_NO_DAEMONIZE env var for running fo…
mikejihbe Oct 3, 2016
42c9874
Updates based on PR feedback: https://github.com/apache/spark/pull/15…
mikejihbe Oct 11, 2016
7010b28
Re-add quotes to env vars. Add the new SPARK_NO_DAEMONIZE flag to the…
mikejihbe Oct 14, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 32 additions & 22 deletions sbin/spark-daemon.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
# SPARK_PID_DIR The pid files are stored. /tmp by default.
# SPARK_IDENT_STRING A string representing this instance of spark. $USER by default
# SPARK_NICENESS The scheduling priority for daemons. Defaults to 0.
# SPARK_NO_DAEMONIZE If set, will run the proposed command in the foreground. It will not output a PID file.
##

usage="Usage: spark-daemon.sh [--config <conf-dir>] (start|stop|submit|status) <spark-command> <spark-instance-number> <args...>"
Expand Down Expand Up @@ -122,6 +123,35 @@ if [ "$SPARK_NICENESS" = "" ]; then
export SPARK_NICENESS=0
fi

execute_command() {
local command="$@"
if [ -z ${SPARK_NO_DAEMONIZE+set} ]; then
nohup -- $command >> $log 2>&1 < /dev/null &
newpid="$!"

echo "$newpid" > "$pid"

#Poll for up to 5 seconds for the java process to start
Copy link
Contributor

@nchammas nchammas Oct 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Space after #. (I know it was like this before your PR.)

for i in {1..10}
do
if [[ $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
break
fi
sleep 0.5
done

sleep 2
# Check if the process has died; in that case we'll tail the log so the user can see
if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
echo "failed to launch $command:"
tail -2 "$log" | sed 's/^/ /'
echo "full log in $log"
fi
else
$command
fi
}

run_command() {
mode="$1"
shift
Expand All @@ -146,13 +176,11 @@ run_command() {

case "$mode" in
(class)
nohup nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-class $command "$@" >> "$log" 2>&1 < /dev/null &
newpid="$!"
execute_command nice -n $SPARK_NICENESS ${SPARK_HOME}/bin/spark-class $command $@
Copy link
Contributor

@nchammas nchammas Oct 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If SPARK_HOME contains spaces, this will break. I recommend quoting both SPARK_HOME and SPARK_NICENESS as they were before.

;;

(submit)
nohup nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-submit --class $command "$@" >> "$log" 2>&1 < /dev/null &
newpid="$!"
execute_command nice -n $SPARK_NICENESS bash ${SPARK_HOME}/bin/spark-submit --class $command $@
Copy link
Contributor

@nchammas nchammas Oct 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here: I would quote the SPARK_ environment variables.

;;

(*)
Expand All @@ -161,24 +189,6 @@ run_command() {
;;
esac

echo "$newpid" > "$pid"

#Poll for up to 5 seconds for the java process to start
for i in {1..10}
do
if [[ $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
break
fi
sleep 0.5
done

sleep 2
# Check if the process has died; in that case we'll tail the log so the user can see
if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
echo "failed to launch $command:"
tail -2 "$log" | sed 's/^/ /'
echo "full log in $log"
fi
}

case $option in
Expand Down