Skip to content

Commit ec08097

Browse files
committed
[SPARK-45497][K8S] Add a symbolic link file spark-examples.jar in K8s Docker images
### What changes were proposed in this pull request? This PR aims to add a symbolic link file, `spark-examples.jar`, in the example jar directory. ``` $ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars | tail -n6 total 1620 drwxr-xr-x 1 root root 4096 Oct 11 04:37 . drwxr-xr-x 1 root root 4096 Sep 9 02:08 .. -rw-r--r-- 1 root root 78803 Sep 9 02:08 scopt_2.12-3.7.1.jar -rw-r--r-- 1 root root 1564255 Sep 9 02:08 spark-examples_2.12-3.5.0.jar lrwxrwxrwx 1 root root 29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar ``` ### Why are the changes needed? Like PySpark example (`pi.py`), we can submit the examples without considering the version numbers which was painful before. ``` bin/spark-submit \ --master k8s://$K8S_MASTER \ --deploy-mode cluster \ ... --class org.apache.spark.examples.SparkPi \ local:///opt/spark/examples/jars/spark-examples.jar 10000 ``` The following is the driver pod log. ``` + exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit ... --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi local:///opt/spark/examples/jars/spark-examples.jar 10000 Files local:///opt/spark/examples/jars/spark-examples.jar from /opt/spark/examples/jars/spark-examples.jar to /opt/spark/work-dir/./spark-examples.jar ``` ### Does this PR introduce _any_ user-facing change? No, this is an additional file. ### How was this patch tested? Manually build the docker image and do `ls`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#43324 from dongjoon-hyun/SPARK-45497. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent fdad095 commit ec08097

1 file changed

Lines changed: 1 addition & 0 deletions

File tree

  • resource-managers/kubernetes/docker/src/main/dockerfiles/spark

resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ COPY sbin /opt/spark/sbin
5959
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
6060
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
6161
COPY examples /opt/spark/examples
62+
RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
6263
COPY kubernetes/tests /opt/spark/tests
6364
COPY data /opt/spark/data
6465

0 commit comments

Comments
 (0)