[SPARK-26362][CORE] Remove 'spark.driver.allowMultipleContexts' to disallow multiple creation of SparkContexts

[HyukjinKwon]

What changes were proposed in this pull request?

Multiple SparkContexts are discouraged and it has been warning for last 4 years, see SPARK-4180. It could cause arbitrary and mysterious error cases, see SPARK-2243.

Honestly, I didn't even know Spark still allows it, which looks never officially supported, see SPARK-2243.

I believe It should be good timing now to remove this configuration.

How was this patch tested?

Each doc was manually checked and manually tested:

$ ./bin/spark-shell --conf=spark.driver.allowMultipleContexts=true
...
scala> new SparkContext()
org.apache.spark.SparkException: Only one SparkContext should be running in this JVM (see SPARK-2243).The currently running SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:939)
...
org.apache.spark.SparkContext$.$anonfun$assertNoOtherContextIsRunning$2(SparkContext.scala:2435)
  at scala.Option.foreach(Option.scala:274)
  at org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2432)
  at org.apache.spark.SparkContext$.markPartiallyConstructed(SparkContext.scala:2509)
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:80)
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:112)
  ... 49 elided

[HyukjinKwon]

adding @srowen, @JoshRosen, @rxin

[srowen]

Honestly I think we can remove this. It's been bad practice for years, and keeping the support means it stays in Spark for years. This mode doesn't really work.

[HyukjinKwon]

Yea, I actually wanted to remove this but made it deprecated in case some people have a different view. +1 for just removing out. Let me update it tomorrow if there's no comment against just removing out.

[rxin]

+1 on removing it.

…

On Thu, Dec 13, 2018 at 8:00 AM, Hyukjin Kwon < ***@***.*** > wrote: Yea, I actually wanted to remove this but made it deprecated in case some people have a different view. +1 for just removing out. Let me update it tomorrow if there's no comment against just removing out. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub ( #23311 (comment) ) , or mute the thread ( https://github.com/notifications/unsubscribe-auth/AATvPB8S8zqiOC60JGg8DD72EFHs8M4kks5u4nmtgaJpZM4ZR4oS ).

[HyukjinKwon]

[error]  * method setActiveContext(org.apache.spark.SparkContext,Boolean)Unit in object org.apache.spark.SparkContext does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SparkContext.setActiveContext")
[error]  * method markPartiallyConstructed(org.apache.spark.SparkContext,Boolean)Unit in object org.apache.spark.SparkContext does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SparkContext.markPartiallyConstructed")

Both are not public APIs. I'm going to exclude this in MiMa.

[gatorsmile]

cc @jiangxb1987

[HeartSaVioR]

+1 on remove it, and IMHO might be better to deprecate it in active 2.x version line.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #100126 has finished for PR 23311 at commit 6482e14.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #100131 has finished for PR 23311 at commit 281cc38.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #100142 has finished for PR 23311 at commit 281cc38.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master.

[jiangxb1987]

late LGTM :)