Commit ec08097
committed
[SPARK-45497][K8S] Add a symbolic link file
### What changes were proposed in this pull request?
This PR aims to add a symbolic link file, `spark-examples.jar`, in the example jar directory.
```
$ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars | tail -n6
total 1620
drwxr-xr-x 1 root root 4096 Oct 11 04:37 .
drwxr-xr-x 1 root root 4096 Sep 9 02:08 ..
-rw-r--r-- 1 root root 78803 Sep 9 02:08 scopt_2.12-3.7.1.jar
-rw-r--r-- 1 root root 1564255 Sep 9 02:08 spark-examples_2.12-3.5.0.jar
lrwxrwxrwx 1 root root 29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar
```
### Why are the changes needed?
Like PySpark example (`pi.py`), we can submit the examples without considering the version numbers which was painful before.
```
bin/spark-submit \
--master k8s://$K8S_MASTER \
--deploy-mode cluster \
...
--class org.apache.spark.examples.SparkPi \
local:///opt/spark/examples/jars/spark-examples.jar 10000
```
The following is the driver pod log.
```
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit ...
--deploy-mode client
--properties-file /opt/spark/conf/spark.properties
--class org.apache.spark.examples.SparkPi
local:///opt/spark/examples/jars/spark-examples.jar 10000
Files local:///opt/spark/examples/jars/spark-examples.jar from /opt/spark/examples/jars/spark-examples.jar to /opt/spark/work-dir/./spark-examples.jar
```
### Does this PR introduce _any_ user-facing change?
No, this is an additional file.
### How was this patch tested?
Manually build the docker image and do `ls`.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes apache#43324 from dongjoon-hyun/SPARK-45497.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>spark-examples.jar in K8s Docker images1 parent fdad095 commit ec08097
1 file changed
Lines changed: 1 addition & 0 deletions
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| 62 | + | |
62 | 63 | | |
63 | 64 | | |
64 | 65 | | |
| |||
0 commit comments