Skip to content

[SPARK-11653][Deploy] Allow spark-daemon.sh to run in the foreground #15338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mikejihbe wants to merge 3 commits into from
Closed

[SPARK-11653][Deploy] Allow spark-daemon.sh to run in the foreground #15338

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cb89755
[SPARK-11653][Deploy] Add a SPARK_NO_DAEMONIZE env var for running fo…
mikejihbe Oct 3, 2016
42c9874
Updates based on PR feedback: https://github.com/apache/spark/pull/15…
mikejihbe Oct 11, 2016
7010b28
Re-add quotes to env vars. Add the new SPARK_NO_DAEMONIZE flag to the…
mikejihbe Oct 14, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 32 additions & 22 deletions sbin/spark-daemon.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
# SPARK_PID_DIR The pid files are stored. /tmp by default.
# SPARK_IDENT_STRING A string representing this instance of spark. $USER by default
# SPARK_NICENESS The scheduling priority for daemons. Defaults to 0.
# SPARK_NO_DAEMONIZE If set, will run the proposed command in the foreground. It will not output a PID file.
##

usage="Usage: spark-daemon.sh [--config <conf-dir>] (start|stop|submit|status) <spark-command> <spark-instance-number> <args...>"
Expand Down Expand Up @@ -122,6 +123,35 @@ if [ "$SPARK_NICENESS" = "" ]; then
export SPARK_NICENESS=0
fi

execute_command() {
command="$@"
Copy link
Member

@jodersky jodersky Oct 11, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark scripts require bash as interpreter, so we can make this a local variable

if [ "$SPARK_NO_DAEMONIZE" != "" ]; then
Copy link
Member

@jodersky jodersky Oct 11, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only checks if the variable is non-empty. Setting it to the empty string will not work. E.g. export SPARK_NO_DAEMONIZE='' will not satisfy the condition.

Something like if [ -z ${SPARK_NO_DAEMONIZE+set} ]; will also handle the null-case (see this SO thread). Another, more elegant, solution would be to use the [[ -v SPARK_NO_DAEMONIZE]], however that requires at least bash 4.2 and is probably not available on all systems that spark runs on.

eval $command
Copy link
Member

@jodersky jodersky Oct 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think eval is required here

else
eval "nohup $command >> \"$log\" 2>&1 < /dev/null &"
Copy link
Member

@jodersky jodersky Oct 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

newpid="$!"

echo "$newpid" > "$pid"

#Poll for up to 5 seconds for the java process to start
Copy link
Contributor

@nchammas nchammas Oct 14, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Space after #. (I know it was like this before your PR.)

for i in {1..10}
do
if [[ $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
break
fi
sleep 0.5
done

sleep 2
# Check if the process has died; in that case we'll tail the log so the user can see
if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
echo "failed to launch $command:"
tail -2 "$log" | sed 's/^/ /'
echo "full log in $log"
fi
fi
}

run_command() {
mode="$1"
shift
Expand All @@ -146,13 +176,11 @@ run_command() {

case "$mode" in
(class)
nohup nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-class $command "$@" >> "$log" 2>&1 < /dev/null &
newpid="$!"
execute_command "nice -n \"$SPARK_NICENESS\" \"${SPARK_HOME}/bin/spark-class\" $command $@"
Copy link
Member

@jodersky jodersky Oct 11, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the outer-most quotes aren't required without the eval.

execute_command nice -n "$SPARK_NICENESS" "${SPARK_HOME}/bin/spark-class" $command $@ should do the trick

;;

(submit)
nohup nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-submit --class $command "$@" >> "$log" 2>&1 < /dev/null &
newpid="$!"
execute_command "nice -n \"$SPARK_NICENESS\" \"${SPARK_HOME}/bin/spark-submit\" --class $command $@"
Copy link
Member

@jodersky jodersky Oct 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

;;

(*)
Expand All @@ -161,24 +189,6 @@ run_command() {
;;
esac

echo "$newpid" > "$pid"

#Poll for up to 5 seconds for the java process to start
for i in {1..10}
do
if [[ $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
break
fi
sleep 0.5
done

sleep 2
# Check if the process has died; in that case we'll tail the log so the user can see
if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
echo "failed to launch $command:"
tail -2 "$log" | sed 's/^/ /'
echo "full log in $log"
fi
}

case $option in
Expand Down