[SPARK-52010] Do not generate API docs for internal classes #50797

cloud-fan · 2025-05-06T01:52:12Z

What changes were proposed in this pull request?

While reviewing the API doc of 4.0 RC4, I found that there are some newly added internal classes appear in the API doc. This PR fixes them.

Why are the changes needed?

API doc should not include internal classes

Does this PR introduce any user-facing change?

no

How was this patch tested?

N/A

Was this patch authored or co-authored using generative AI tooling?

no

cloud-fan · 2025-05-06T01:52:36Z

common/utils/src/main/scala/org/apache/spark/util/SparkStringUtils.scala

 package org.apache.spark.util

-trait SparkStringUtils {
+private[spark] trait SparkStringUtils {


added in #49062

cloud-fan · 2025-05-06T01:52:50Z

common/utils/src/main/scala/org/apache/spark/util/SparkTestUtils.scala

 import scala.jdk.CollectionConverters._

-trait SparkTestUtils {
+private[spark] trait SparkTestUtils {


added in #43354

cloud-fan · 2025-05-06T01:55:08Z

connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala

 import org.apache.spark.sql.errors.QueryCompilationErrors
 import org.apache.spark.sql.types._

-private[sql] case class AvroDataToCatalyst(


4.0 added two new internal classes in this package: AvroExpressionEvalUtils, AvroCompressionCodec. Given that this package contains only internal classes and deprecated APIs since 3.0, I marked the entire org.apache.spark.sql.avro package as private, so that we no longer need private[sql] in these existing classes.

cloud-fan · 2025-05-06T01:55:42Z

project/SparkBuild.scala

-        "org.apache.spark.sql.scripting"
+        "org.apache.spark.kafka010",
+        "org.apache.spark.network",
+        "org.apache.spark.sql.avro",


See https://github.com/apache/spark/pull/50797/files#r2074591645

cloud-fan · 2025-05-06T01:56:55Z

.../kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaConfigUpdater.scala

 /**
 * Class to conveniently update Kafka config params, while logging the changes
 */
-private[spark] case class KafkaConfigUpdater(module: String, kafkaParams: Map[String, Object])


4.0 added one new internal class in this package: KafkaTokenProviderExceptions. Given that this package contains only internal classes, I marked the entire org.apache.spark.kafka010 package as private, so that we no longer need private[spark] in these existing classes.

cloud-fan · 2025-05-06T01:57:09Z

project/SparkBuild.scala

        "org.apache.spark.deploy",
-        "org.apache.spark.util.collection",
-        "org.apache.spark.sql.scripting"
+        "org.apache.spark.kafka010",


See https://github.com/apache/spark/pull/50797/files#r2074596294

cloud-fan · 2025-05-06T01:57:30Z

project/SparkBuild.scala

+        "org.apache.spark.network",
+        "org.apache.spark.sql.avro",
+        "org.apache.spark.sql.scripting",
+        "org.apache.spark.types.variant",


The new variant project adds this new package, but it only contain internal classes.

cloud-fan · 2025-05-06T01:57:59Z

project/SparkBuild.scala

+        "org.apache.spark.sql.avro",
+        "org.apache.spark.sql.scripting",
+        "org.apache.spark.types.variant",
+        "org.apache.spark.ui.flamegraph",


added by #42988 , should be intrenal

cloud-fan · 2025-05-06T01:58:34Z

sql/catalyst/src/main/scala/org/apache/spark/sql/exceptions/SqlScriptingException.scala

 import org.apache.spark.sql.exceptions.SqlScriptingException.errorMessageWithLineNumber

-class SqlScriptingException (
+private[sql] class SqlScriptingException (


added by the new scripting feature.

cloud-fan · 2025-05-06T01:58:52Z

sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala

 * Make the `classifyException` method throw out the original exception
 */
-trait NoLegacyJDBCError extends JdbcDialect {
+private[sql] trait NoLegacyJDBCError extends JdbcDialect {


added by #46937

cloud-fan · 2025-05-06T01:59:34Z

cc @HyukjinKwon @dongjoon-hyun @yaooqinn @hvanhovell

dongjoon-hyun

+1, it looks good to me. Thank you, @cloud-fan .

### What changes were proposed in this pull request? While reviewing the API doc of 4.0 RC4, I found that there are some newly added internal classes appear in the API doc. This PR fixes them. ### Why are the changes needed? API doc should not include internal classes ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes #50797 from cloud-fan/api. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit 9f5ae88) Signed-off-by: Kent Yao <[email protected]>

yaooqinn · 2025-05-06T03:51:07Z

Merged to master/4.0, thank you @cloud-fan @dongjoon-hyun

HyukjinKwon

LGTM2

### What changes were proposed in this pull request? This is a followup of #50797 , to exclde these private packages from mima check. ### Why are the changes needed? mima shouldn't check private APIs ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes #51038 from cloud-fan/mima. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? This is a followup of #50797 , to exclde these private packages from mima check. ### Why are the changes needed? mima shouldn't check private APIs ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes #51038 from cloud-fan/mima. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit f634311) Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? While reviewing the API doc of 4.0 RC4, I found that there are some newly added internal classes appear in the API doc. This PR fixes them. ### Why are the changes needed? API doc should not include internal classes ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#50797 from cloud-fan/api. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Kent Yao <[email protected]>

### What changes were proposed in this pull request? This is a followup of apache#50797 , to exclde these private packages from mima check. ### Why are the changes needed? mima shouldn't check private APIs ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#51038 from cloud-fan/mima. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? While reviewing the API doc of 4.0 RC4, I found that there are some newly added internal classes appear in the API doc. This PR fixes them. ### Why are the changes needed? API doc should not include internal classes ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#50797 from cloud-fan/api. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit 3ebc0fd) Signed-off-by: Kent Yao <[email protected]>

### What changes were proposed in this pull request? This is a followup of apache#50797 , to exclde these private packages from mima check. ### Why are the changes needed? mima shouldn't check private APIs ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#51038 from cloud-fan/mima. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 2bed378) Signed-off-by: Wenchen Fan <[email protected]>

Do not generate API docs for private classes

9452d5d

github-actions bot added SQL BUILD DSTREAM AVRO labels May 6, 2025

cloud-fan commented May 6, 2025

View reviewed changes

cloud-fan changed the title ~~[SPARK-52010] Do not generate API docs for private classes~~ [SPARK-52010] Do not generate API docs for internal classes May 6, 2025

dongjoon-hyun approved these changes May 6, 2025

View reviewed changes

yaooqinn approved these changes May 6, 2025

View reviewed changes

yaooqinn closed this in 9f5ae88 May 6, 2025

HyukjinKwon reviewed May 6, 2025

View reviewed changes

cloud-fan mentioned this pull request May 28, 2025

[INFRA][MINOR] Update mima exclude #51038

Closed

[SPARK-52010] Do not generate API docs for internal classes #50797

[SPARK-52010] Do not generate API docs for internal classes #50797

Uh oh!

Conversation

cloud-fan commented May 6, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

cloud-fan May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented May 6, 2025

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

yaooqinn commented May 6, 2025

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cloud-fan May 6, 2025 •

edited

Loading