Skip to content

0.5.0

Choose a tag to compare

View all tags
@dongjoon-hyun dongjoon-hyun released this 02 Oct 15:15
· 66 commits to main since this release
0.5.0
0d47cfe
This commit was signed with the committer’s verified signature.
dongjoon-hyun Dongjoon Hyun
GPG key ID: EDA00CE834F0FC5C
Verified
Learn about vigilant mode.

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.5.0 (App Version) with Helm chart 1.3.0 (Chart Version).

Notable Features

Docker Image

docker pull apache/spark-kubernetes-operator:0.5.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list
NAME   NAMESPACE  REVISION UPDATED                            	STATUS   CHART                           APP VERSION
spark  default    1        2025-06-04 11:06:24.61408 -0700 PDT  deployed spark-kubernetes-operator-1.3.0 0.5.0

Full Changelog:

Resolved Issues

  • SPARK-52604 Release Spark Kubernetes Operator 0.5.0 (v1)
  • SPARK-52605 Bump version to 0.5.0-SNAPSHOT in main branch
  • SPARK-52606 Promote Spark CRDs to v1
  • SPARK-52607 Add pi-v1beta1.yaml example
  • SPARK-52608 Make README.md up-to-date
  • SPARK-52623 Add spark.ui.port to Spark History Server example
  • SPARK-52624 Upgrade spotbugs-tool to 4.9.3 and spotbugs-plugin to 6.1.13
  • SPARK-52645 Enable ExitOnOutOfMemoryError JVM option by default
  • SPARK-52647 Fix jvmArgs to support multiple arguments
  • SPARK-52648 Add support for maximal retain duration for Spark application resources
  • SPARK-52653 Fix *.operatorContainer to *.operatorPod.operatorContainer in operations.md
  • SPARK-52654 Use ParallelGC by default
  • SPARK-52657 Allow spark.app.id
  • SPARK-52661 Suppress pending pod log messages
  • SPARK-52662 Add SparkApp benchmark
  • SPARK-52664 Add branch-0.4 to publish_snapshot_*.yml GitHub Action jobs
  • SPARK-52667 Improve SparkAppReconciler to show app names during cleaning up
  • SPARK-52668 Set (Initial|Max)RAMPercentage to 80% by default
  • SPARK-52676 Use Java 21 for SparkApp benchmark
  • SPARK-52679 Revise appender.console.layout.pattern to include MDC keys and thread name
  • SPARK-52702 Add test for comparing generated CRD yaml with the version in Helm charts
  • SPARK-52754 Fix repeated status update retries if no error
  • SPARK-52755 Remove ConfigMap informer from config reconciler
  • SPARK-52773 Upgrade Gradle to 8.14.3
  • SPARK-52774 Fix resource retain policy typo in docs
  • SPARK-52827 Add pi-with-template.yaml example
  • SPARK-52831 Revise the default logging layout to include only name & namespace MDC Keys
  • SPARK-52834 Improve ProbeService to show the port number
  • SPARK-52835 Fix the default value of operatorContainer memory to 2Gi from 4Gi
  • SPARK-52836 Fix sparkapp.sh to let Spark driver determine pod memory
  • SPARK-52915 Support TTL for Spark apps
  • SPARK-52997 Fixes wrong worker assignment if multiple clusters are deployed to the same namespace
  • SPARK-53026 Fix deserialize error for spec with DriverServiceIngressSpec field
  • SPARK-53027 Spark Operator service account should be able to create ingress
  • SPARK-53159 Declare metrics port for operator in helm chart
  • SPARK-53186 Fix probe port override from helm chart
  • SPARK-53187 Support SparkCluster event related metrics set
  • SPARK-53245 Update Java Operator SDK version to 5.1.2
  • SPARK-53325 Support Prometheus 2.0 text-based-format and best practices for metrics naming
  • SPARK-53405 Add metrics recording for latency of Spark app state transition
  • SPARK-53407 Use baseDirectory instead of baseDir
  • SPARK-53409 Upgrade Gradle to 9.0.0
  • SPARK-53410 Improve merge_spark_pr.py to accept PR numbers as a CLI argument
  • SPARK-53411 Upgrade the minimum K8s version to v1.32
  • SPARK-53430 Prevent unintended JSON serialization with @JsonIgnore
  • SPARK-53462 Upgrade actions/checkout to v5
  • SPARK-53468 Update setup-minikube to v0.0.20
  • SPARK-53493 Add pi-preview.yaml example
  • SPARK-53495 Remove unused 'persistentVolume' access from operator rbac
  • SPARK-53508 Upgrade Spark to 4.0.1
  • SPARK-53510 Update SparkClusterResourceSpec to use apache/spark:4.0.1 as the default value of container image
  • SPARK-53511 Make examples directory up-to-date with Spark 4.0.1
  • SPARK-53513 Update YuniKorn docs with 1.7.0
  • SPARK-53542 Update CI to test K8s 1.34
  • SPARK-53545 Enforce style check at project level for operator
  • SPARK-53588 Upgrade kubernetes-client to 7.4.0
  • SPARK-53607 Support Java 25
  • SPARK-53613 Upgrade google-java-format to 1.28.0 to support Java 25
  • SPARK-53617 Use zulu Java distribution in GitHub Action jobs
  • SPARK-53624 Use bitnamisecure/kubectl:latest for Helm Chart testing
  • SPARK-53627 Update docs to recommend K8s 1.32+
  • SPARK-53628 Upgrade pmd and spotbugs and enable checking on Java 25
  • SPARK-53639 Use spark consistently for release-name of Helm installation
  • SPARK-53642 Improve Javadoc for interfaces, class, methods, variables
  • SPARK-53644 Upgrade Gradle to 9.1.0
  • SPARK-53646 Improve KubernetesMetricsInterceptorTest to verify http.request metric
  • SPARK-53647 Use io.fabric8.kubernetes.client.http.Interceptor instead of okhttp3.Interceptor
  • SPARK-53648 Use VertxHttpClientFactory instead of OkHttpClientFactory
  • SPARK-53649 Remove logging-interceptor dependency
  • SPARK-53650 Make build.gradle and libs.versions.toml up-to-date with okhttp3 usage
  • SPARK-53663 Add spark-history-server-preview.yaml
  • SPARK-53664 Update updateResponseMetrics to handle valid responses only
  • SPARK-53669 Publish SBOM artifacts
  • SPARK-53670 Use Gradle Java Toolchain
  • SPARK-53679 Fix typos in Spark Kubernetes Operator documentation
  • SPARK-53701 Use Java 25 toolchain while supporting Java 17+
  • SPARK-53702 Update tests to use 4.0.1 and 3.5.7
  • SPARK-53703 Improve unit tests to be neutral from Spark versions
  • SPARK-53704 Rename pyspark-pi.yaml to pi-python.yaml
  • SPARK-53706 App reconcile steps should properly handle exceptions in status update
  • SPARK-53710 Upgrade junit to 5.13.4
  • SPARK-53711 Upgrade operator-sdk to 5.1.3
  • SPARK-53712 Fix spark-operator to use JUnit 5 instead of JUnit 4
  • SPARK-53713 Remove org.jetbrains.annotations test code dependency from spark-operator
  • SPARK-53714 Remove okhttp3 dependency
  • SPARK-53721 Fix HelmChart template to use Spark-version agnostic SentinelResource
  • SPARK-53722 Document workloadResources.sparkApplicationSentinel.name
  • SPARK-53723 Support SparkCluster Sentinel Resource
  • SPARK-53727 Update GitHub Action YAML file in branch-0.5
Assets 4
Loading