[SPARK-33006][K8S][DOCS] Add dynamic PVC usage example into K8s doc #29897
Conversation
dongjoon-hyun
changed the title
|
Test build #129197 has finished for PR 29897 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
| The configuration properties for mounting volumes into the executor pods use prefix `spark.kubernetes.executor.` instead of `spark.kubernetes.driver.`. For a complete list of available options for each supported type of volumes, please refer to the [Spark Properties](#spark-properties) section below. | ||
| The configuration properties for mounting volumes into the executor pods use prefix `spark.kubernetes.executor.` instead of `spark.kubernetes.driver.`. | ||
|
|
||
| For example, you can mount a dynamically-created persistent volume claim per executor by using `OnDemand` as a claim name and `storageClass` and `sizeLimit` options like the following. This is useful in case of [Dynamic Allocation](configuration.html#dynamic-allocation). |
There was a problem hiding this comment.
Can you explain why it's useful for DA? Currently, this doesn't support DA yet. I think the use-case will be for people to use larger shuffle disks in Kube where larger local disks are not allowed.
There was a problem hiding this comment.
@dbtsai . What do you mean by the following?
Currently, this doesn't support DA yet.
Since Apache Spark 3.0.0, dynamic allocation with K8s has been supported with shuffle data tracking. And, this feature is also developed for both additional large disk requirement and dynamic allocation scenario. For example, in case of dynamic allocation, the executor id increases monotonically and indefinitely, so users cannot prepare unlimited pre-populated PVCs. With this feature, the PVC is created and deleted dynamically with the same lifecycle with the executor pod.
There was a problem hiding this comment.
In short, I made the PVC(and PV) be created at every pod creation in ExecutorPodsAllocator.scala. Please see the original code. For example, if you kill some pod, new executor is going to be created with a new PVC.
- [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors #29846 [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors
cc @viirya
viirya
left a comment
viirya
left a comment
There was a problem hiding this comment.
Looks okay after clarifying Dynamic Allocation case.
|
Please see my comment, @viirya ~ |
4 participants
What changes were proposed in this pull request?
This updates K8s document to describe new dynamic PVC features.
Why are the changes needed?
This will help the user use the new features easily.
Does this PR introduce any user-facing change?
Yes, but it's a doc updates.
How was this patch tested?
Manual.