Skip to content

Releases: apache/spark-kubernetes-operator

0.6.0

07 Nov 18:42
@dongjoon-hyun dongjoon-hyun
0.6.0
25ca83f
This commit was signed with the committer’s verified signature.
dongjoon-hyun Dongjoon Hyun
GPG key ID: EDA00CE834F0FC5C
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.6.0 Latest
Latest

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.6.0 (App Version) with Helm chart 1.4.0 (Chart Version).

Notable Features

Docker Image

docker pull apache/spark-kubernetes-operator:0.6.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list
NAME   NAMESPACE  REVISION UPDATED                              STATUS   CHART                           APP VERSION
spark  default    1        2025-11-07 10:31:35.82152 -0800 PST	deployed spark-kubernetes-operator-1.4.0 0.6.0

Full Changelog:

Resolved Issues

  • SPARK-53725 Bump version to 0.6.0-SNAPSHOT in main branch
  • SPARK-53726 Release Spark Kubernetes Operator 0.6.0
  • SPARK-53739 Add pi-preview-with-eventlog.yaml example
  • SPARK-53740 Update (pi|spark-history-server)-preview.yaml to use 4.1.0-preview2
  • SPARK-53767 Upgrade Dropwizard metrics to 4.2.33
  • SPARK-53768 Upgrade lombok to 1.18.42
  • SPARK-53769 Enable Annotation Processing during Java compilation
  • SPARK-53770 Upgrade mockito to 5.20.0
  • SPARK-53778 Use JDK25 Gradle Image as builder images
  • SPARK-53780 spark-submission-worker should depend on io.fabric8 directly
  • SPARK-53781 Exclude Spark's transitive dependencies consistently across modules
  • SPARK-53782 Use slf4j-api as a direct dependency
  • SPARK-53783 Use log4j2 instead of log4j
  • SPARK-53787 Upgrade Spark to 4.1.0-preview2
  • SPARK-53813 Recover Publish Snapshot Chart GitHub Action Job by using GitHash
  • SPARK-53814 Use Java List.of instead of Collections.(empty|singleton)List
  • SPARK-53815 Remove branch-0.4 from daily publish_snapshot_* GitHub Action jobs
  • SPARK-53817 Use o.a.s.util.Pair instead of o.a.commons.lang3.tuple.Pair
  • SPARK-53818 Improve SentinelManager.toString by JEP-280 instead of ToStringBuilder
  • SPARK-53820 Introduce o.a.s.k8s.operator.utils.StringUtils
  • SPARK-53821 Ban org.apache.commons.lang3 package
  • SPARK-53824 Ban org.apache.commons.collections4 package
  • SPARK-53825 Use Java MessageDigest instead of org.apache.commons.codec
  • SPARK-53826 Use Java 25 in GitHub Action jobs consistently
  • SPARK-53827 Exclude commons-(codec|compress) and com.ibm.icu transitive dependencies
  • SPARK-53828 Exclude org/apache/spark/ui/static/** from Spark operator jar
  • SPARK-53830 Exclude .idea directory from Spotless check
  • SPARK-53851 Set io.netty.noUnsafe to true to avoid JEP-498 warnings
  • SPARK-53852 Add spark.kubernetes.driver.pod.excludedFeatureSteps usage example
  • SPARK-53853 Add Example section in operators.md
  • SPARK-53863 Remove Java 17 requirement from deploy.gradle
  • SPARK-53869 Support multiple files in pyFiles
  • SPARK-53871 Upgrade JUnit to 6.0.0
  • SPARK-53872 Revisit JUnit assert usage in test cases
  • SPARK-53874 SparkAppDriverConf should respect sparkVersion of SparkApplication CRD
  • SPARK-53899 Update Operator SDK to 5.1.4
  • SPARK-53909 Fix RBAC to allow Spark driver to create StatefulSet
  • SPARK-53910 Add StatefulSet-based SparkApp example
  • SPARK-53911 Support SPARK_VERSION placeholder in container image names
  • SPARK-53912 Add cluster-preview.yaml and spark-connect-server-preview.yaml
  • SPARK-53932 Add FAIR schedule examples with Spark (Connect|Thrift) Servers
  • SPARK-53933 Add Apache Iceberg example
  • SPARK-53983 Increase s.k.o.reconciler.foregroundRequestTimeoutSeconds to 60s
  • SPARK-53984 Use CRD v1 instead of v1beta1 in benchmark script
  • SPARK-53985 Suppress StatusRecorder warning messages
  • SPARK-53987 Upgrade JaCoCo to 0.8.14 for official Java 25 support
  • SPARK-53988 Upgrade Spotless to 8.0.0
  • SPARK-53989 Optimize sparkapps.sh to use kubectl delete --all
  • SPARK-53990 Use Java (Map|Set).of instead of Collections.(empty|singleton)(Set|Map)
  • SPARK-53992 Add SparkOperatorConfManager.getAll method
  • SPARK-53993 Support spark.logConf configuration
  • SPARK-53994 Exclude KerberosConfDriverFeatureStep during benchmarking
  • SPARK-54010 Support restart counter in Spark app
  • SPARK-54070 Add spark.logConf configuration to operator docs
  • SPARK-54073 Improve ConfOptionDocGenerator to generate a sorted doc by config key
  • SPARK-54081 Add word-count-preview.yaml Example
  • SPARK-54082 Upgrade Spark to 4.1.0-preview3
  • SPARK-54092 Use Java-friendly KubernetesClientUtils APIs
  • SPARK-54097 Upgrade Gradle to 9.2.0
  • SPARK-54107 Use 4.1.0-preview3-java21-scala image for preview examples
  • SPARK-54120 Update assertGeneratedCRDMatchesHelmChart to include diff
  • SPARK-54167 Update GitHub Action YAML file in branch-0.6
Assets 4
Loading

0.5.0

02 Oct 15:15
@dongjoon-hyun dongjoon-hyun
0.5.0
0d47cfe
This commit was signed with the committer’s verified signature.
dongjoon-hyun Dongjoon Hyun
GPG key ID: EDA00CE834F0FC5C
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.5.0

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.5.0 (App Version) with Helm chart 1.3.0 (Chart Version).

Notable Features

Docker Image

docker pull apache/spark-kubernetes-operator:0.5.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list
NAME   NAMESPACE  REVISION UPDATED                            	STATUS   CHART                           APP VERSION
spark  default    1        2025-06-04 11:06:24.61408 -0700 PDT  deployed spark-kubernetes-operator-1.3.0 0.5.0

Full Changelog:

Resolved Issues

  • SPARK-52604 Release Spark Kubernetes Operator 0.5.0 (v1)
  • SPARK-52605 Bump version to 0.5.0-SNAPSHOT in main branch
  • SPARK-52606 Promote Spark CRDs to v1
  • SPARK-52607 Add pi-v1beta1.yaml example
  • SPARK-52608 Make README.md up-to-date
  • SPARK-52623 Add spark.ui.port to Spark History Server example
  • SPARK-52624 Upgrade spotbugs-tool to 4.9.3 and spotbugs-plugin to 6.1.13
  • SPARK-52645 Enable ExitOnOutOfMemoryError JVM option by default
  • SPARK-52647 Fix jvmArgs to support multiple arguments
  • SPARK-52648 Add support for maximal retain duration for Spark application resources
  • SPARK-52653 Fix *.operatorContainer to *.operatorPod.operatorContainer in operations.md
  • SPARK-52654 Use ParallelGC by default
  • SPARK-52657 Allow spark.app.id
  • SPARK-52661 Suppress pending pod log messages
  • SPARK-52662 Add SparkApp benchmark
  • SPARK-52664 Add branch-0.4 to publish_snapshot_*.yml GitHub Action jobs
  • SPARK-52667 Improve SparkAppReconciler to show app names during cleaning up
  • SPARK-52668 Set (Initial|Max)RAMPercentage to 80% by default
  • SPARK-52676 Use Java 21 for SparkApp benchmark
  • SPARK-52679 Revise appender.console.layout.pattern to include MDC keys and thread name
  • SPARK-52702 Add test for comparing generated CRD yaml with the version in Helm charts
  • SPARK-52754 Fix repeated status update retries if no error
  • SPARK-52755 Remove ConfigMap informer from config reconciler
  • SPARK-52773 Upgrade Gradle to 8.14.3
  • SPARK-52774 Fix resource retain policy typo in docs
  • SPARK-52827 Add pi-with-template.yaml example
  • SPARK-52831 Revise the default logging layout to include only name & namespace MDC Keys
  • SPARK-52834 Improve ProbeService to show the port number
  • SPARK-52835 Fix the default value of operatorContainer memory to 2Gi from 4Gi
  • SPARK-52836 Fix sparkapp.sh to let Spark driver determine pod memory
  • SPARK-52915 Support TTL for Spark apps
  • SPARK-52997 Fixes wrong worker assignment if multiple clusters are deployed to the same namespace
  • SPARK-53026 Fix deserialize error for spec with DriverServiceIngressSpec field
  • SPARK-53027 Spark Operator service account should be able to create ingress
  • SPARK-53159 Declare metrics port for operator in helm chart
  • SPARK-53186 Fix probe port override from helm chart
  • SPARK-53187 Support SparkCluster event related metrics set
  • SPARK-53245 Update Java Operator SDK version to 5.1.2
  • SPARK-53325 Support Prometheus 2.0 text-based-format and best practices for metrics naming
  • SPARK-53405 Add metrics recording for latency of Spark app state transition
  • SPARK-53407 Use baseDirectory instead of baseDir
  • SPARK-53409 Upgrade Gradle to 9.0.0
  • SPARK-53410 Improve merge_spark_pr.py to accept PR numbers as a CLI argument
  • SPARK-53411 Upgrade the minimum K8s version to v1.32
  • SPARK-53430 Prevent unintended JSON serialization with @JsonIgnore
  • SPARK-53462 Upgrade actions/checkout to v5
  • SPARK-53468 Update setup-minikube to v0.0.20
  • SPARK-53493 Add pi-preview.yaml example
  • SPARK-53495 Remove unused 'persistentVolume' access from operator rbac
  • SPARK-53508 Upgrade Spark to 4.0.1
  • SPARK-53510 Update SparkClusterResourceSpec to use apache/spark:4.0.1 as the default value of container image
  • SPARK-53511 Make examples directory up-to-date with Spark 4.0.1
  • SPARK-53513 Update YuniKorn docs with 1.7.0
  • SPARK-53542 Update CI to test K8s 1.34
  • SPARK-53545 Enforce style check at project level for operator
  • SPARK-53588 Upgrade kubernetes-client to 7.4.0
  • SPARK-53607 Support Java 25
  • SPARK-53613 Upgrade google-java-format to 1.28.0 to support Java 25
  • SPARK-53617 Use zulu Java distribution in GitHub Action jobs
  • SPARK-53624 Use bitnamisecure/kubectl:latest for Helm Chart testing
  • SPARK-53627 Update docs to recommend K8s 1.32+
  • SPARK-53628 Upgrade pmd and spotbugs and enable checking on Java 25
  • SPARK-53639 Use spark consistently for release-name of Helm installation
  • SPARK-53642 Improve Javadoc for interfaces, class, methods, variables
  • SPARK-53644 Upgrade Gradle to 9.1.0
  • SPARK-53646 Improve KubernetesMetricsInterceptorTest to verify http.request metric
  • SPARK-53647 Use io.fabric8.kubernetes.client.http.Interceptor instead of okhttp3.Interceptor
  • SPARK-53648 Use VertxHttpClientFactory instead of OkHttpClientFactory
  • SPARK-53649 Remove logging-interceptor dependency
  • SPARK-53650 Make build.gradle and libs.versions.toml up-to-date with okhttp3 usage
  • SPARK-53663 Add spark-history-server-preview.yaml
  • SPARK-53664 Update updateResponseMetrics to handle valid responses only
  • SPARK-53669 Publish SBOM artifacts
  • SPARK-53670 Use Gradle Java Toolchain
  • SPARK-53679 Fix typos in Spark Kubernetes Operator documentation
  • SPARK-53701 Use Java 25 toolchain while supporting Java 17+
  • SPARK-53702 Update tests to use 4.0.1 and 3.5.7
  • SPARK-53703 Improve unit tests to be neutral from Spark versions
  • SPARK-53704 Rename pyspark-pi.yaml to pi-python.yaml
  • SPARK-53706 App reconcile steps should properly handle exceptions in status update
  • SPARK-53710 Upgrade junit to 5.13.4
  • SPARK-53711 Upgrade operator-sdk to 5.1.3
  • SPARK-53712 Fix spark-operator to use JUnit 5 instead of JUnit 4
  • SPARK-53713 Remove org.jetbrains.annotations test code dependency from spark-operator
  • SPARK-53714 Remove okhttp3 dependency
  • SPARK-53721 Fix HelmChart template to use Spark-version agnostic SentinelResource
  • SPARK-53722 Document workloadResources.sparkApplicationSentinel.name
  • SPARK-53723 Support SparkCluster Sentinel Resource
  • SPARK-53727 Update GitHub Action YAML file in branch-0.5
Loading

0.4.0

03 Jul 16:21
@dongjoon-hyun dongjoon-hyun
0.4.0
3879459
This commit was signed with the committer’s verified signature.
dongjoon-hyun Dongjoon Hyun
GPG key ID: EDA00CE834F0FC5C
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
0.4.0

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.4.0 (App Version) with Helm chart 1.2.0 (Chart Version).

Notable Features

Docker Image

docker pull apache/spark-kubernetes-operator:0.4.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list
NAME   NAMESPACE  REVISION UPDATED                            	STATUS   CHART                           APP VERSION
spark  default    1        2025-06-04 11:06:24.61408 -0700 PDT  deployed spark-kubernetes-operator-1.2.0 0.4.0

Full Changelog:

Resolved Issues

[SPARK-52378] - Release Spark Kubernetes Operator 0.4.0
[SPARK-52086] - Upgrade operator-sdk to 5.1.1
[SPARK-52379] - Bump version to 0.4.0-SNAPSHOT in main branch
[SPARK-52416] - Upgrade Gradle to 8.14.2
[SPARK-52435] - Update CIs to use actions/checkout@v4 consistently
[SPARK-52464] - Update .asf.yaml with new README.md link
[SPARK-52466] - Clean up unnecessary comments in e2e tests
[SPARK-52467] - Add dfs-read-write and localstack examples
[SPARK-52471] - Add Swift-based SparkPi K8s CronJob example
[SPARK-52474] - Limit GHA job execution time to up to 20 minutes
[SPARK-52477] - Upgrade JUnit to 5.13.1 and Mockito to 5.18.0
[SPARK-52478] - Use GCS Maven Central mirror as the primary repository
[SPARK-52479] - Upgrade the minimum K8s version to v1.31
[SPARK-52480] - Remove scalaVersion from Spark 4+ examples
[SPARK-52481] - Add Spark History Server example
[SPARK-52512] - Add pi-with-eventlog example
[SPARK-52513] - Add a streaming word count example with rolling event logs
[SPARK-52526] - Add Spark Thrift Server example
[SPARK-52555] - Enforce UnusedLocalVariable rule
[SPARK-52558] - Lower SparkOperatorConfManager log level to WARN for FileNotFoundException
[SPARK-52559] - Synchronize SparkOperatorConfManager.getValue
[SPARK-52581] - Revise AppCleanUpStep to include cleanup logic for all states
[SPARK-52602] - Document all public classes
[SPARK-52603] - Update GitHub Action YAML file in branch-0.4

Loading

0.3.0

04 Jun 18:11
@dongjoon-hyun dongjoon-hyun
0.3.0
d9ba7eb

Choose a tag to compare

View all tags
0.3.0

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.3.0 (App Version) with Helm chart 1.1.0 (Chart Version).

Notable Features

Docker Image

docker pull apache/spark-kubernetes-operator:0.3.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list
NAME   NAMESPACE  REVISION UPDATED                            	STATUS   CHART                           APP VERSION
spark  default    1        2025-06-04 11:06:24.61408 -0700 PDT  deployed spark-kubernetes-operator-1.1.0 0.3.0

Full Changelog:

Resolved Issues

Loading

0.2.0

20 May 18:24
@dongjoon-hyun dongjoon-hyun
0.2.0
85f17fb

Choose a tag to compare

View all tags
0.2.0

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.2.0 (App Version) with Helm chart 1.0.0 (Chart Version).

Notable Features

Docker Image

docker pull apache/spark-kubernetes-operator:0.2.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list
NAME 	NAMESPACE	REVISION	UPDATED                            	STATUS  	CHART                          	APP VERSION
spark	default  	1       	2025-05-20 11:39:27.72567 -0700 PDT	deployed	spark-kubernetes-operator-1.0.0	0.2.0

Full Changelog:

Resolved Issues

Loading

v0.1.0

08 May 00:28
@dongjoon-hyun dongjoon-hyun
v0.1.0
fe9ec3a
This commit was signed with the committer’s verified signature.
dongjoon-hyun Dongjoon Hyun
GPG key ID: EDA00CE834F0FC5C
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v0.1.0

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. v0.1.0 is the initial release of Apache Spark K8s Operator.

Notable Features

Docker Image

docker pull apache/spark-kubernetes-operator:0.1.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator

Full Changelog:

Resolved Issues

  • [SPARK-47745] Add License to Spark Operator repository
  • [SPARK-47889] Setup gradle as build tool for operator repository
  • [SPARK-47929] Setup Static Analysis for Operator
  • [SPARK-47943] Add GitHub Action CI for Java Build and Test
  • [SPARK-47950] Add Java API Module for Spark Operator
  • [SPARK-48015] Update build.gradle to fix deprecation warnings
  • [SPARK-48017] Add Spark application submission worker for operator
  • [SPARK-48120] Enable autolink to SPARK jira issue
  • [SPARK-48326] Use the official Apache Spark 4.0.0-preview1
  • [SPARK-48382] Add reconciler to spark-operator module
  • [SPARK-48398] Add Helm Chart
  • [SPARK-48528] Refine K8s Operator merge_spark_pr.py to use kubernetes-operator-x.y.z version
  • [SPARK-48606] Upgrade google-java-format to 1.22.0
  • [SPARK-48679] Upgrade checkstyle and spotbugs version
  • [SPARK-48786] Define Config Loading Framework for Spark Operator Controller
  • [SPARK-48952] Use BasePluginExtension in spark-operator/build.gradle
  • [SPARK-48953] Upgrade Gradle to 8.9
  • [SPARK-48968] Avoid unnecessary task configuration in spark-operator-api
  • [SPARK-48969] Fix spark-operator module to define test framework explicitly
  • [SPARK-48971] Ban org.apache.commons.collections in favor of collections4
  • [SPARK-48984] Add Controller Metrics System and Utils
  • [SPARK-49019] Use try-with-resources to test KubernetesClientFactory.buildKubernetesClient
  • [SPARK-49020] Avoid raw type usage
  • [SPARK-49036] Exclude JUnitAssertionsShouldIncludeMessage/JUnitTestContainsTooManyAsserts PMD rules and simplify test code
  • [SPARK-49045] Add docker image build for operator
  • [SPARK-49052] Add SparkOperator class and tests
  • [SPARK-49129] Fix ENTRYPOINT to point /opt/spark-operator/operator/docker-entrypoint.sh
  • [SPARK-49130] Verify built images in build-image CI job via docker run test
  • [SPARK-49132] Minimize docker image by removing redundant chown commands
  • [SPARK-49144] Use the latest setup-java v4 with cache feature
  • [SPARK-49148] Use the latest PMD 6.x rules instead of the deprecated ones
  • [SPARK-49153] Increase Gradle JVM memory to 4g like Spark repo
  • [SPARK-49156] Enforce ImmutableField and UselessParentheses rules
  • [SPARK-49157] Enforce SignatureDeclareThrowsException and AvoidThrowingRawExceptionTypes rules
  • [SPARK-49158] Enforce ConfusingTernary and PrematureDeclaration rules
  • [SPARK-49159] Enforce FieldDeclarationsShouldBeAtStartOfClass, LinguisticNaming and ClassWithOnlyPrivateConstructorsShouldBeFinal rules
  • [SPARK-49165] Fix RestartPolicyTest to cover SchedulingFailure
  • [SPARK-49167] Enforce UseUtilityClass rule
  • [SPARK-49168] Add OpenContainers Annotations to docker image
  • [SPARK-49214] Add e2e test in operator workflow
  • [SPARK-49236] Update README.md with build/test/CI info
  • [SPARK-49237] Speed up docker image building by excluding check instead of test
  • [SPARK-49238] Upgrade Gradle to 8.10
  • [SPARK-49239] Add k8s-integration-tests GitHub Action CI job
  • [SPARK-49253] Install and test Helm chart in K8s integration test CI
  • [SPARK-49256] Upgrade kubernetes-client to 6.13.3 and commons-lang3 to 3.16.0
  • [SPARK-49258] Add pi.yaml example and update README.md
  • [SPARK-49270] Add sql.yaml example
  • [SPARK-49271] Improve gradlew to support both curl and wget
  • [SPARK-49276] Use API Group spark.apache.org
  • [SPARK-49277] Refactor RestartPolicyTest to test per case
  • [SPARK-49278] Revise reconcilesteps package and SparkAppReconciler
  • [SPARK-49279] Revise observers package
  • [SPARK-49280] Add pi-with-one-pod.yaml example
  • [SPARK-49289] Fix docker-entrypoint.sh to quote the environment variables
  • [SPARK-49290] Remove commons-lang3 dependency from spark-operator-api
  • [SPARK-49291] Fix javadoc generation and add lint test pipeline to prevent
  • [SPARK-49293] Add _MESSAGE postfix to DRIVER_(READY|RUNNING)
  • [SPARK-49295] Add NOTICE, NOTICE-binary, LICENSE-binary files and update *.gradle files
  • [SPARK-49296] Add deploy.gradle to support publish-related tasks
  • [SPARK-49298] Reduce spark-operator fat jar size by excluding dependencies
  • [SPARK-49299] Add buildDockerImage Gradle Task
  • [SPARK-49304] Add -SNAPSHOT postfix to Spark Operator version
  • [SPARK-49305] Revise Spark Operator docker image
  • [SPARK-49309] Use HTTP_* constant variables instead of magic numbers
  • [SPARK-49315] Generalize relocateGeneratedCRD Gradle Task to handle *.spark.apache.org-v1.yml
  • [SPARK-49316] Generalize printer-columns.sh to handle *.spark.apache.org-v1.yml files
  • [SPARK-49317] Add pi-on-yunikorn.yaml example and update README.md
  • [SPARK-49319] Add SparkCluster to spark-operator-api module and examples
  • [SPARK-49322] Add SparkCluster to spark-submission-worker module
  • [SPARK-49324] Add SparkApplication state transition e2e test
  • [SPARK-49325] Add SparkCluster to spark-operator module
  • [SPARK-49328] Propagate Spark configurations to SparkCluster
  • [SPARK-49329] Support user provided spec for SparkCluster
  • [SPARK-49330] Revise InstanceConfig to ExecutorInstanceConfig class
  • [SPARK-49331] Use setup-minikube GitHub Action
  • [SPARK-49332] Add K8s service for Workers to SparkClusterResourceSpec
  • [SPARK-49340] Document SparkCluster and add submit-pi-to-prod.sh example
  • [SPARK-49343] Document nightly versions of operator image and Helm Chart
  • [SPARK-49346] Add publish_snapshot_dockerhub.yml Daily GitHub Action job
  • [SPARK-49348] Support schedulerName for SparkCluster
  • [SPARK-49351] Add publish_snapshot_chart.yml GitHub Action job
  • [SPARK-49360] Use rsync to upload to nightlies
  • [SPARK-49361] Fix Dockerfile by removing unused ARG from builder and moving default value
  • [SPARK-49362] Simplify snapshot HelmChart to use apache/spark-kubernetes-operator:main-snapshot by default
  • [SPARK-49376] Introduce ClusterToleration and WorkerInstanceConfig
  • [SPARK-49377] Fix e2e catch step application not found
  • [SPARK-49379] Enable Pull Request Labeler
  • [SPARK-49380] Add cluster-with-template.yaml example
  • [SPARK-49388] Remove SPARK_NO_DAEMONIZE in favor of live log UIs
  • [SPARK-49389] Support master|worker container templates
  • [SPARK-49395] Add 8081 port to Worker resource spec
  • [SPARK-49397] Add Clean Up section to README.md
  • [SPARK-49399] Add pi-scala.yaml and pyspark-pi.yaml
  • [SPARK-49400] Use Gradle Version Catalog
  • [SPARK-49403] Add log4j2 default setting to values.yaml
  • [SPARK-49404] Adjust ERROR-level log messages
  • [SPARK-49464] Add documentations
  • [SPARK-49468] Add Hot Properties Reload Test
  • [SPARK-49514] Fix K8s version in GitHub Action CI
  • [SPARK-49516] Upgrade the minimum K8s version to v1.28
  • [SPARK-49527] Add ConfOptionDocGenerator to generate Spark Operator Config Property Doc
  • [SPARK-49590] Fix javaVersion to jdkVersion in E2E spark-versions test template
  • [SPARK-49619] Upgrade Gradle to 8.10.1
  • [SPARK-49623] Refactor prefix appResources to workloadResources
  • [SPARK-49625] Add SparkCluster state transition test
  • [SPARK-49645] Update e2e/python/chainsaw-test.yaml to use non-R image
  • [SPARK-49657] Add multi instances e2e
  • [SPARK-49658] Refactor e2e tests pipelines
  • [SPARK-49705] Use spark-examples.jar instead of spark-examples_2.13-4.0.0-preview1.jar
  • [SPARK-49706] Use apache/spark images instead of spark
  • [SPARK-49714] Add Java 21-based SparkPi example
  • [SPARK-49715] Add Java 21-based SparkCluster example
  • [SPARK-49724] Add java21 to the e2e tests
  • [SPARK-49735] Upgrade Spark to 4.0.0-preview2
  • [SPARK-49742] Upgrade README, examples, tests to use preview2
  • [SPARK-49754] Support HPA for SparkCluster
  • [SPARK-49778] Remove (master|worker) prefix from field names of (Master|Worker)Spec
  • [SPARK-49790] Support HPA template for SparkCluster
  • [SPARK-49818] Update E2E tests to use Spark 3.5.3
  • [SPARK-49830] Add required fields for sparkApplicationSentinel
  • [SPARK-49831] Provide empty RuntimeVersions object to ClusterSpec.runtimeVersions by default
  • [SPARK-49832] Make o.a.s.k8s.operator.utils.Utils argument naming consistent
  • [SPARK-49838] Add spark-version label to Spark Cluster resources
  • [SPARK-49887] Update cluster-with-template.yaml example with pod annotation
  • [SPARK-49888] Use registry.k8s.io/pause:3.9 to avoid failure deterministically
  • [SPARK-50305] Upgrade Gradle to 8.11
  • [SPARK-50384] Update docs with YuniKorn 1.6.0 and first-time installation guide
  • [SPARK-50401] Upgrade kubernetes-client to 6.13.4 and log4j to 2.24.2
    ...
Read more
Loading