Bump Spark 3.5.4 #11731

pan3793 · 2024-12-09T07:04:32Z

Bump Spark 3.5.4, with minimal necessary changes related to SPARK-50235 and SPARK-50463.

Note: this PR targets to 1.7.2 and 1.6.2 (for Java8 users)

pan3793 · 2024-12-10T02:46:23Z

the upgrading causes a core dump in iceberg-spark-3.5's tests

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8529a34563, pid=83775, tid=83793
#
# JRE version: OpenJDK Runtime Environment Zulu17.54+21-CRaC-CA (17.0.13+11) (build 17.0.13+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.54+21-CRaC-CA (17.0.13+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xc34563]  ProtectionDomainEntry::object_no_keepalive()+0x3
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/work/iceberg/iceberg/spark/v3.5/spark/core.83775)
#
# An error report file with more information is saved as:
# /home/runner/work/iceberg/iceberg/spark/v3.5/spark/hs_err_pid83775.log
#
# Compiler replay data is saved as:
# /home/runner/work/iceberg/iceberg/spark/v3.5/spark/replay_pid83775.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#
> Task :iceberg-spark:iceberg-spark-3.5_2.12:test FAILED

pan3793 · 2024-12-10T03:14:09Z

the failure seems related to arrow

minimal reproduce command

./gradlew -DsparkVersions=3.5 -DscalaVersion=2.12 -DhiveVersions= -DflinkVersions= -DkafkaVersions= \
    :iceberg-spark:iceberg-spark-3.5_2.12:test --tests=TestRewriteDataFilesAction \
    -Pquick=true -x javadoc

pan3793 · 2024-12-10T15:51:02Z

update: the failure is related to SPARK-50235, and the test passed on Spark 3.5.4 RC1 with reverting that patch

viirya · 2024-12-10T17:24:40Z

Please check for some discussion there apache/spark#49131 (comment)

LuciferYang · 2024-12-10T18:13:43Z

Please check for some discussion there apache/spark#49131 (comment)

Yes, I tested it locally and found that after adding a no-op closeIfFreeable method to the IcebergArrowColumnVector.java in version 3.5, all the test cases in TestRewriteDataFilesAction were able to pass. We can further test it after the release of 3.5.4 RC2.

pan3793 · 2024-12-11T02:16:32Z

@viirya @LuciferYang thank you for the guidance, let me try

pan3793 · 2024-12-11T02:39:37Z

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java

+  // If a column vector is writable or constant, it should override this method and do nothing.
+  // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
+  @Override
+  public void closeIfNotWritable() {}


will rename to closeIfFreeable after RC2 available

If a column vector is writable or constant, it should override this method and do nothing.

we have ConstantColumnVector as well, should we do it vector classes which absolutely requires it ?

I checked the codebase, other vectors overwrite void close() {}, so they are not affected

singhpk234

Nice catch @pan3793 !

singhpk234 · 2024-12-11T03:15:17Z

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java

+  // If a column vector is writable or constant, it should override this method and do nothing.
+  // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
+  @Override
+  public void closeIfNotWritable() {}


If a column vector is writable or constant, it should override this method and do nothing.

we have ConstantColumnVector as well, should we do it vector classes which absolutely requires it ?

pan3793 · 2024-12-11T04:21:27Z

FYI @LuciferYang @viirya @dongjoon-hyun @wForget

CI is green now, and I think the current Spark branch-3.5 is in good shape for Iceberg.

dongjoon-hyun · 2024-12-11T16:07:29Z

Thank you for checking Apache Iceberg side, @pan3793 .

Fokko · 2024-12-17T14:54:39Z

Thanks @pan3793 for catching this early. Could you create a separate PR for just the patch? We can include this in 1.7.2.

pan3793 · 2024-12-17T16:07:41Z

@Fokko I opened #11802

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java

pan3793 · 2024-12-20T12:33:34Z

Spark 3.5.4 RC3 passed the vote and the jars were available on Maven Central a few minutes ago, I removed the staging repo and it's ready to go.

cc @nastra @Fokko and @jbonofre

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java

…ectorized/IcebergArrowColumnVector.java

singhpk234

LGTM as well, thanks @pan3793 !

Fokko · 2025-01-03T20:02:30Z

Thanks @pan3793 for fixing this! And thanks @singhpk234, @huaxingao, @LuciferYang, and @viirya for reviewing!

pan3793 · 2025-03-03T09:15:50Z

@Fokko @nastra, May I request that release 1.6.2 ship this patch to allow the users who are stuck with Java 8 to use the latest Spark 3.5.x with Iceberg?

gunrk · 2025-05-21T17:00:51Z

@pan3793 Can you confirm if this was ever patched to 1.6.x? I don't see a 1.6.2 release.
We face similar segv errors in our spark executors after upgrading to spark 3.5.4 with iceberg 1.6.1

# A fatal error has been detected by the Java Runtime Environment:
--
# SIGSEGV (0xb) at pc=0x00007f047c7f1511, pid=1, tid=0x00007ef4069e9700

We see IlegalAccessError and the process exits with code 139.
Unfortunately we are on Java 8 and It seems Iceberg 1.7 removed support for the same.

pan3793 · 2025-05-22T05:53:05Z

@gunrk no 1.6.2 yet, I think you can request a release on the mailing list.

github-actions bot added the build label Dec 9, 2024

pan3793 mentioned this pull request Dec 10, 2024

[SPARK-50463][SQL][3.5] Fix ConstantColumnVector with Columnar to Row conversion apache/spark#49131

Closed

dongjoon-hyun mentioned this pull request Dec 10, 2024

[SPARK-50235][SQL] Clean up ColumnVector resource after processing all rows in ColumnarToRowExec apache/spark#48767

Closed

github-actions bot added the spark label Dec 11, 2024

pan3793 commented Dec 11, 2024

View reviewed changes

singhpk234 reviewed Dec 11, 2024

View reviewed changes

pan3793 force-pushed the spark-3.5.4 branch from 9144835 to 99bcc1c Compare December 16, 2024 09:11

pan3793 changed the title ~~Bump Spark 3.5.4 RC1~~ Bump Spark 3.5.4 RC2 Dec 16, 2024

pan3793 added 3 commits December 17, 2024 21:55

Bump Spark 3.5.4 RC1

c3d0b8a

3.5.4 RC2

18bb711

Spark 3.5.4 RC3

2feb6b0

pan3793 force-pushed the spark-3.5.4 branch from dfdc640 to 2feb6b0 Compare December 17, 2024 13:55

LuciferYang mentioned this pull request Dec 17, 2024

Spark 3.5: Adapt to Spark 3.5.4 #11802

Closed

nastra reviewed Dec 17, 2024

View reviewed changes

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java Outdated Show resolved Hide resolved

pan3793 added 2 commits December 18, 2024 10:28

comment

0c25054

release

7bb2100

pan3793 changed the title ~~Bump Spark 3.5.4 RC2~~ Bump Spark 3.5.4 Dec 20, 2024

pan3793 marked this pull request as ready for review December 20, 2024 12:31

Fokko added this to the Iceberg 1.7.2 milestone Dec 20, 2024

Fokko reviewed Dec 20, 2024

View reviewed changes

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java Outdated Show resolved Hide resolved

pan3793 commented Dec 20, 2024

View reviewed changes

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java Outdated Show resolved Hide resolved

Update spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/v…

dc7aea5

…ectorized/IcebergArrowColumnVector.java

huaxingao approved these changes Dec 20, 2024

View reviewed changes

viirya approved these changes Dec 20, 2024

View reviewed changes

LuciferYang approved these changes Dec 21, 2024

View reviewed changes

Fokko approved these changes Dec 21, 2024

View reviewed changes

singhpk234 approved these changes Dec 24, 2024

View reviewed changes

Fokko merged commit dbfefb0 into apache:main Jan 3, 2025
49 checks passed

Fokko pushed a commit that referenced this pull request Jan 3, 2025

Bump Apache Spark to 3.5.4 (#11731)

5a4e4e2

Fokko mentioned this pull request Jan 6, 2025

Bump pyspark from 3.5.3 to 3.5.4 apache/iceberg-python#1461

Closed

Fokko mentioned this pull request Jan 27, 2025

Test out Iceberg 1.7.2 RC3 apache/iceberg-python#1581

Closed

jackylee-ch mentioned this pull request Feb 3, 2025

Bump Spark version to 3.5.4 apache/incubator-gluten#8653

Closed

pan3793 mentioned this pull request Feb 6, 2025

Incorrect Results and SIGSEGV on Read with Iceberg + PySpark + Nessie #12178

Closed

3 tasks

This was referenced Mar 4, 2025

Bump Spark version from 3.5.2 to 3.5.5 apache/incubator-gluten#8889

Closed

[GLUTEN-8889][CORE] Bump Spark version from 3.5.2 to 3.5.5 apache/incubator-gluten#8890

Merged

jackylee-ch mentioned this pull request Mar 18, 2025

[GLUTEN-8889][CORE] Bump Spark version from 3.5.2 to 3.5.5 for CH backend apache/incubator-gluten#9031

Closed

Kontinuation mentioned this pull request May 22, 2025

Spark executors failing occasionally on SIGSEGV apache/datafusion-comet#1714

Closed

Bump Spark 3.5.4 #11731

Bump Spark 3.5.4 #11731

Uh oh!

Conversation

pan3793 commented Dec 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Dec 10, 2024

Uh oh!

pan3793 commented Dec 10, 2024

Uh oh!

pan3793 commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viirya commented Dec 10, 2024

Uh oh!

LuciferYang commented Dec 10, 2024

Uh oh!

pan3793 commented Dec 11, 2024

Uh oh!

pan3793 Dec 11, 2024

Choose a reason for hiding this comment

Uh oh!

singhpk234 Dec 11, 2024

Choose a reason for hiding this comment

Uh oh!

pan3793 Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

singhpk234 Dec 11, 2024

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Dec 11, 2024

Uh oh!

dongjoon-hyun commented Dec 11, 2024

Uh oh!

Fokko commented Dec 17, 2024

Uh oh!

pan3793 commented Dec 17, 2024

Uh oh!

Uh oh!

pan3793 commented Dec 20, 2024

Uh oh!

Uh oh!

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko commented Jan 3, 2025

Uh oh!

pan3793 commented Mar 3, 2025

Uh oh!

gunrk commented May 21, 2025

Uh oh!

pan3793 commented May 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

pan3793 commented Dec 9, 2024 •

edited

Loading

pan3793 commented Dec 10, 2024 •

edited

Loading

pan3793 Dec 11, 2024 •

edited

Loading