Skip to content

spark-core - [SPARK-4787] - Stop sparkcontext properly if a DAGScheduler init error occurs #3809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tigerquoll wants to merge 2 commits into from
Closed

spark-core - [SPARK-4787] - Stop sparkcontext properly if a DAGScheduler init error occurs #3809

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2172578
[SPARK-4787] Stop context properly if an exception occurs during DAGS…
tigerquoll Dec 26, 2014
5661e01
[SPARK-4787] Ensure that call to stop() doesn't lose the exception by…
tigerquoll Dec 30, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,8 +329,11 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
try {
dagScheduler = new DAGScheduler(this)
} catch {
case e: Exception => throw
new SparkException("DAGScheduler cannot be initialized due to %s".format(e.getMessage))
case e: Exception => {
stop()
Copy link
Contributor

@JoshRosen JoshRosen Dec 29, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, do you think this should be in a try-finally block so that we don't swallow the useful "DAGScheduler could not be initialized" exception if the stop() call somehow fails?

Copy link
Contributor Author

@tigerquoll tigerquoll Dec 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent idea Josh.

throw
Copy link
Contributor

@JoshRosen JoshRosen Dec 26, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style nit: you can use string interpolation instead of String.format, which will allow the new SparkException to fit on the same line as throw:

throw new SparkException(s"DAGScheduler cannot be initialized due to ${e.getMessage}")

However, I'd prefer to call the two-argument constructor which takes the cause as second argument, since this will lead to more informative stacktraces:

throw new SparkException("Error while constructing DAGScheduler", e)

new SparkException("DAGScheduler cannot be initialized due to %s".format(e.getMessage))
}
}

// start TaskScheduler after taskScheduler sets DAGScheduler reference in DAGScheduler's
Expand Down