[SPARK-42219][CORE] Introducing a config to close all active SparkContexts after the Main method has finished

[attilapiros]

What changes were proposed in this pull request?

Introducing a config to close all active SparkContexts after the Main method has finished.

Why are the changes needed?

We run into errors after upgrading from Spark 3.1 to Spark 3.2 as the SparkContext get closed right after the starting of the application. It turned out the root cause is SPARK-34674 which introduced the closing of the SparkContexts after the Main method has finished. For details see #32283.

This application was a spark job server built on top of springboot so all the job submits were outside of the main method.

Does this PR introduce any user-facing change?

With the current default (true) so it is the same behaviour as for YARN.

How was this patch tested?

Manually.

[attilapiros]

cc @dongjoon-hyun, @HyukjinKwon

[dongjoon-hyun]

That's a bad example from ancient YARN age, @attilapiros . I'd not re-enforce those bad habits. Instead, I can give you the counter examples like

spark/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

Line 308 in aeb2a13

sparkConf.getBoolean("spark.kubernetes.submitInDriver", false)

spark/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala

Lines 88 to 90 in aeb2a13

	if (sparkConf.getOption("spark.yarn.appMasterEnv.OMP_NUM_THREADS").isEmpty &&
	sparkConf.getOption("spark.mesos.driverEnv.OMP_NUM_THREADS").isEmpty &&
	sparkConf.getOption("spark.kubernetes.driverEnv.OMP_NUM_THREADS").isEmpty) {

[attilapiros]

@dongjoon-hyun I have moved the config into the k8s module

Stale.

[attilapiros]

cc @holdenk

[attilapiros]

This pyspark test failure is unrelated:

Error:  running /__w/spark/spark/python/run-tests --modules=pyspark-core,pyspark-streaming,pyspark-ml --parallelism=1 ; received return code 255
Error: Process completed with exit code 19.

[attilapiros]

cc @HyukjinKwon, @srowen

[pan3793]

SPARK-42698(#40314) is aiming to expand the scope of stopping SparkContext after runMain exiting to all cluster managers, if it is accepted, the configuration proposed by this PR should expand scope as well.

[srowen]

Should this be specific to Kubernetes?
Does it need to be a config or a method you can call?
Actually, why would you not kill the contexts after main exits in any case?

[attilapiros]

@srowen

Should this be specific to Kubernetes?

The original #32283 was Kubernetes specific. This PR just adds a new config to have the old behaviour as default but make the new one also available.

Does it need to be a config or a method you can call?

Unfortunately there is a use case for both behaviour. See the next point.

Actually, why would you not kill the contexts after main exits in any case?

I bumped into this change when I analysed an application where spark was used as a job server.
With the older spark it was running just fine but after an update it was stopping in the very beginning.
The app was built on top of springboot where job requests was served via REST.
In a springboot app the main method just initialises / registers the REST handlers and the serving of the new requests done on separate threads. With the new behaviour the Spark context was closed right after the initialisation.

[attilapiros]

cc @mridulm

[zhouyifan279]

Our customer also encountered this issue recently. They are migrating their spark job server (a spring boot application) from Spark 2.4 to Spark 3.
They feel surprised that only Spark 3.1.2 stops SparkContext after job server started.

[attilapiros]

I am closing this PR. Job servers has the option to do blocking call in the main method to avoid the auto stopping of the active spark contexts.