Skip to content

[HWORKS-1372] Apply uniffle shuffle service and close sparkContext after main method has finished #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SirOibaf merged 5 commits into from
Jun 23, 2024
Merged

[HWORKS-1372] Apply uniffle shuffle service and close sparkContext after main method has finished #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b80888c
Apply patches for uniffle
gibchikafa Nov 21, 2023
881ad7b
Update kubernetes client version
gibchikafa Apr 2, 2024
0d742dc
[SPARK-34674][CORE][K8S] Close SparkContext after the Main method has…
kotlovs Apr 22, 2021
c995f7c
Update version
gibchikafa Jun 20, 2024
b52aa31
Fix update version
gibchikafa Jun 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/kvstore/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/network-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/network-shuffle/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/network-yarn/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/sketch/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/tags/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/unsafe/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ private[spark] class ExecutorAllocationManager(
// decommissioning without a shuffle service.
if (conf.get(config.DYN_ALLOCATION_SHUFFLE_TRACKING_ENABLED) ||
(decommissionEnabled &&
conf.get(config.STORAGE_DECOMMISSION_SHUFFLE_BLOCKS_ENABLED))) {
conf.get(config.STORAGE_DECOMMISSION_SHUFFLE_BLOCKS_ENABLED) || conf.isRssEnable())) {
logWarning("Dynamic allocation without a shuffle service is an experimental feature.")
} else if (!testing) {
throw new SparkException("Dynamic allocation of executors requires the external " +
Expand Down
2 changes: 2 additions & 0 deletions core/src/main/scala/org/apache/spark/SparkConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,8 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging with Seria
this
}

def isRssEnable(): Boolean = get("spark.shuffle.manager", "sort").contains("RssShuffleManager")

private[spark] def set[T](entry: ConfigEntry[T], value: T): SparkConf = {
set(entry.key, entry.stringConverter(value))
this
Expand Down
9 changes: 9 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -952,6 +952,15 @@ private[spark] class SparkSubmit extends Logging {
} catch {
case t: Throwable =>
throw findCause(t)
} finally {
if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
!isThriftServer(args.mainClass)) {
try {
SparkContext.getActive.foreach(_.stop())
} catch {
case e: Throwable => logError(s"Failed to close SparkContext: $e")
}
}
}
}

Expand Down
3 changes: 2 additions & 1 deletion core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2030,7 +2030,8 @@ private[spark] class DAGScheduler(
// if the cluster manager explicitly tells us that the entire worker was lost, then
// we know to unregister shuffle output. (Note that "worker" specifically refers to the process
// from a Standalone cluster, where the shuffle service lives in the Worker.)
val fileLost = workerHost.isDefined || !env.blockManager.externalShuffleServiceEnabled
val fileLost = (workerHost.isDefined || !env.blockManager.externalShuffleServiceEnabled) &&
!sc.getConf.isRssEnable()
removeExecutorAndUnregisterOutputs(
execId = execId,
fileLost = fileLost,
Expand Down
3 changes: 2 additions & 1 deletion core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -973,7 +973,8 @@ private[spark] class TaskSetManager(
// and we are not using an external shuffle server which could serve the shuffle outputs.
// The reason is the next stage wouldn't be able to fetch the data from this dead executor
// so we would need to rerun these tasks on other executors.
if (isShuffleMapTasks && !env.blockManager.externalShuffleServiceEnabled && !isZombie) {
if (isShuffleMapTasks && !env.blockManager.externalShuffleServiceEnabled && !isZombie
&& !conf.isRssEnable()) {
for ((tid, info) <- taskInfos if info.executorId == execId) {
val index = taskInfos(tid).index
// We may have a running task whose partition has been marked as successful,
Expand Down
45 changes: 25 additions & 20 deletions dev/deps/spark-deps-hadoop-2.7-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,26 +155,31 @@ jsr305/3.0.0//jsr305-3.0.0.jar
jta/1.1//jta-1.1.jar
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
kubernetes-client/4.12.0//kubernetes-client-4.12.0.jar
kubernetes-model-admissionregistration/4.12.0//kubernetes-model-admissionregistration-4.12.0.jar
kubernetes-model-apiextensions/4.12.0//kubernetes-model-apiextensions-4.12.0.jar
kubernetes-model-apps/4.12.0//kubernetes-model-apps-4.12.0.jar
kubernetes-model-autoscaling/4.12.0//kubernetes-model-autoscaling-4.12.0.jar
kubernetes-model-batch/4.12.0//kubernetes-model-batch-4.12.0.jar
kubernetes-model-certificates/4.12.0//kubernetes-model-certificates-4.12.0.jar
kubernetes-model-common/4.12.0//kubernetes-model-common-4.12.0.jar
kubernetes-model-coordination/4.12.0//kubernetes-model-coordination-4.12.0.jar
kubernetes-model-core/4.12.0//kubernetes-model-core-4.12.0.jar
kubernetes-model-discovery/4.12.0//kubernetes-model-discovery-4.12.0.jar
kubernetes-model-events/4.12.0//kubernetes-model-events-4.12.0.jar
kubernetes-model-extensions/4.12.0//kubernetes-model-extensions-4.12.0.jar
kubernetes-model-metrics/4.12.0//kubernetes-model-metrics-4.12.0.jar
kubernetes-model-networking/4.12.0//kubernetes-model-networking-4.12.0.jar
kubernetes-model-policy/4.12.0//kubernetes-model-policy-4.12.0.jar
kubernetes-model-rbac/4.12.0//kubernetes-model-rbac-4.12.0.jar
kubernetes-model-scheduling/4.12.0//kubernetes-model-scheduling-4.12.0.jar
kubernetes-model-settings/4.12.0//kubernetes-model-settings-4.12.0.jar
kubernetes-model-storageclass/4.12.0//kubernetes-model-storageclass-4.12.0.jar
kubernetes-client-api/6.11.0//kubernetes-client-api-6.11.0.jar
kubernetes-client/6.11.0//kubernetes-client-6.11.0.jar
kubernetes-httpclient-okhttp/6.11.0//kubernetes-httpclient-okhttp-6.11.0.jar
kubernetes-model-admissionregistration/6.11.0//kubernetes-model-admissionregistration-6.11.0.jar
kubernetes-model-apiextensions/6.11.0//kubernetes-model-apiextensions-6.11.0.jar
kubernetes-model-apps/6.11.0//kubernetes-model-apps-6.11.0.jar
kubernetes-model-autoscaling/6.11.0//kubernetes-model-autoscaling-6.11.0.jar
kubernetes-model-batch/6.11.0//kubernetes-model-batch-6.11.0.jar
kubernetes-model-certificates/6.11.0//kubernetes-model-certificates-6.11.0.jar
kubernetes-model-common/6.11.0//kubernetes-model-common-6.11.0.jar
kubernetes-model-coordination/6.11.0//kubernetes-model-coordination-6.11.0.jar
kubernetes-model-core/6.11.0//kubernetes-model-core-6.11.0.jar
kubernetes-model-discovery/6.11.0//kubernetes-model-discovery-6.11.0.jar
kubernetes-model-events/6.11.0//kubernetes-model-events-6.11.0.jar
kubernetes-model-extensions/6.11.0//kubernetes-model-extensions-6.11.0.jar
kubernetes-model-flowcontrol/6.11.0//kubernetes-model-flowcontrol-6.11.0.jar
kubernetes-model-gatewayapi/6.11.0//kubernetes-model-gatewayapi-6.11.0.jar
kubernetes-model-metrics/6.11.0//kubernetes-model-metrics-6.11.0.jar
kubernetes-model-networking/6.11.0//kubernetes-model-networking-6.11.0.jar
kubernetes-model-node/6.11.0//kubernetes-model-node-6.11.0.jar
kubernetes-model-policy/6.11.0//kubernetes-model-policy-6.11.0.jar
kubernetes-model-rbac/6.11.0//kubernetes-model-rbac-6.11.0.jar
kubernetes-model-resource/6.11.0//kubernetes-model-resource-6.11.0.jar
kubernetes-model-scheduling/6.11.0//kubernetes-model-scheduling-6.11.0.jar
kubernetes-model-storageclass/6.11.0//kubernetes-model-storageclass-6.11.0.jar
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
libfb303/0.9.3//libfb303-0.9.3.jar
libthrift/0.12.0//libthrift-0.12.0.jar
Expand Down
2 changes: 1 addition & 1 deletion examples/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/avro/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/docker-integration-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kafka-0-10-assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kafka-0-10-sql/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kafka-0-10-token-provider/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kafka-0-10/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kinesis-asl-assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kinesis-asl/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/spark-ganglia-lgpl/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion graphx/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hadoop-cloud/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion launcher/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion mllib-local/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion mllib/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
3 changes: 2 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<packaging>pom</packaging>
<name>Spark Project Parent POM</name>
<url>http://spark.apache.org/</url>
Expand Down Expand Up @@ -207,6 +207,7 @@
<arrow.version>2.0.0</arrow.version>
<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
<leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
<kubernetes-client.version>6.11.0</kubernetes-client.version>

<javax.servlet-api.name>javax.servlet-api</javax.servlet-api.name>

Expand Down
2 changes: 1 addition & 1 deletion repl/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/integration-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion resource-managers/mesos/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion resource-managers/yarn/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.12</artifactId>
<version>3.1.1.5</version>
<version>3.1.1.6</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
Loading