[SPARK-47307][SQL] Add a config to optionally chunk base64 strings

[wForget]

Follow up #45408

What changes were proposed in this pull request?

[SPARK-47307] Add a config to optionally chunk base64 strings

Why are the changes needed?

In #35110, it was incorrectly asserted that:

ApacheCommonBase64 obeys http://www.ietf.org/rfc/rfc2045.txt

This is not true as the previous code called:

public static byte[] encodeBase64(byte[] binaryData)

Which states:

Encodes binary data using the base64 algorithm but does not chunk the output.

However, the RFC 2045 (MIME) base64 encoder does chunk by default. This now means that any Spark encoded base64 strings cannot be decoded by encoders that do not implement RFC 2045. The docs state RFC 4648.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing test suite.

Was this patch authored or co-authored using generative AI tooling?

No

[yaooqinn]

Why don't we use the encoder directly?

[wForget]

Why don't we use the encoder directly?

java.util.Base64$Encoder is not serializable.

-- !query
select base64(c7), base64(c8), base64(v), ascii(s) from char_tbl4
-- !query schema
struct<>
-- !query output
java.io.NotSerializableException
java.util.Base64$Encoder
Serialization stack:
	- object not serializable (class: java.util.Base64$Encoder, value: java.util.Base64$Encoder@423ed07f)
	- element of array (index: 2)
	- array (class [Ljava.lang.Object;, size 5)
	- field (class: org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory, name: org$apache$spark$sql$execution$WholeStageCodegenEvaluatorFactory$$references, type: class [Ljava.lang.Object;)
	- object (class org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory, org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory@2fd9633e)
	- element of array (index: 0)
	- array (class [Ljava.lang.Object;, size 1)
	- field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
	- object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=class org.apache.spark.sql.execution.WholeStageCodegenExec, functionalInterfaceMethod=scala/Function2.apply:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;, implementation=invokeStatic org/apache/spark/sql/execution/WholeStageCodegenExec.$anonfun$doExecute$4$adapted:(Lorg/apache/spark/sql/execution/WholeStageCodegenEvaluatorFactory;Ljava/lang/Object;Lscala/collection/Iterator;)Lscala/collection/Iterator;, instantiatedMethodType=(Ljava/lang/Object;Lscala/collection/Iterator;)Lscala/collection/Iterator;, numCaptured=1])
	- writeReplace data (class: java.lang.invoke.SerializedLambda)
	- object (class org.apache.spark.sql.execution.WholeStageCodegenExec$$Lambda$2458/0x000002cf3e949c30, org.apache.spark.sql.execution.WholeStageCodegenExec$$Lambda$2458/0x000002cf3e949c30@603a0fa7)

[yaooqinn]

maybe we can follow the work I have done for Encode to make it RuntimeReplaceable with StaticInvoke

[wForget]

maybe we can follow the work I have done for Encode to make it RuntimeReplaceable with StaticInvoke

Thanks, I will try it

[yaooqinn]

nit: BASE_64 to BASE64

[gatorsmile]

@wForget is it true?

@yaooqinn @dongjoon-hyun @wForget I think we need to discuss it in the dev list before merging this PR. This will break the encryption that relies on base64.

I would suggest reverting it first before making a common decision on the dev list.

[dongjoon-hyun]

Thank you for the feedback, @gatorsmile.

I have three questions to understand your request.

• Do you see the example, SPARK-47307? What is your opinion?
• We have the following migration guide, do you want to set spark.sql.legacy.chunkBase64String.enabled=true by default in Apache Spark 3.5.2?

Since 3.5.2, the base64 function will return a non-chunked string. To restore the behavior of chunking base64 encoded strings into lines of at most 76 characters, set spark.sql.legacy.chunkBase64String.enabled to true.

• If we change the default value, do we still need to revert?

[cloud-fan]

I think the PR description should be improved to mention more information. It's a behavior change, we should clearly define the behavior difference and measure the impact. Does it mean the Spark base64 result can be decoded by more encoders now? Is there any standard encoder that may fail to decode after this change?

[wForget]

Sorry, I didn't change PR description after changing default value for new config.
Actually, a behavior change was unexpectedly introduced in #35110, this PR aims to make the behavior of base64 encode configurable. If the change in current behavior is controversial, I would first submit a PR to change default value of the newly introduced config which would revert to the previous behavior. WDYT?

[yaooqinn]

I would first submit a PR to change default value of the newly introduced config which would revert to the previous behavior.
+1