[SPARK-49152][SQL] V2SessionCatalog should use V2Command #47724

amaliujia · 2024-08-12T19:23:28Z

What changes were proposed in this pull request?

V2SessionCatalog should use V2Command when possible.

Why are the changes needed?

This is because the session catalog can be overwritten thus the overwritten's catalog should use v2 commands, otherwise the V1Command will still call hive metastore or the built-in session catalog.

Does this PR introduce any user-facing change?

NO

How was this patch tested?

Existing tests.

Was this patch authored or co-authored using generative AI tooling?

NO

V2SessionCatalog should use V2Command when possible. This is because the session catalog can be overwritten thus the overwritten's catalog should use v2 commands, otherwise the V1Command will still call hive metastore or the built-in session catalog. No Existing tests. NO Closes apache#47660 from amaliujia/create_table_v2. Authored-by: Rui Wang <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

amaliujia · 2024-08-12T19:23:47Z

@cloud-fan

cloud-fan · 2024-08-12T23:33:46Z

sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala


    case ShowTableExtended(
-        DatabaseInSessionCatalog(db),
+      ResolvedV1Database(db),


Suggested change

ResolvedV1Database(db),

ResolvedV1Database(db),

…/ResolveSessionCatalog.scala

cloud-fan · 2024-08-13T03:06:10Z

thanks, merging to 3.5!

### What changes were proposed in this pull request? V2SessionCatalog should use V2Command when possible. ### Why are the changes needed? This is because the session catalog can be overwritten thus the overwritten's catalog should use v2 commands, otherwise the V1Command will still call hive metastore or the built-in session catalog. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? Existing tests. ### Was this patch authored or co-authored using generative AI tooling? NO Closes #47724 from amaliujia/branch-3.5. Lead-authored-by: Rui Wang <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

manuzhang · 2024-09-19T03:42:48Z

sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala


  object ResolvedV1Identifier {
+    def unapply(resolved: LogicalPlan): Option[TableIdentifier] = resolved match {
+      case ResolvedIdentifier(catalog, ident) if supportsV1Command(catalog) =>


@amaliujia @cloud-fan
This change looks to have broken creating V1 table from a V2_SESSION_CATALOG_IMPLEMENTATION like Iceberg's SparkSessionCatalog

spark/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala

Lines 159 to 170 in e693e18

// For CREATE TABLE [AS SELECT], we should use the v1 command if the catalog is resolved to the

// session catalog and the table provider is not v2.

case c @ CreateTable(ResolvedV1Identifier(ident), _, _, tableSpec: TableSpec, _) =>

val (storageFormat, provider) = getStorageFormatAndProvider(

c.tableSpec.provider, tableSpec.options, c.tableSpec.location, c.tableSpec.serde,

ctas = false)

if (!isV2Provider(provider)) {

constructV1TableCmd(None, c.tableSpec, ident, c.tableSchema, c.partitioning,

c.ignoreIfExists, storageFormat, provider)

} else {

c

}

Does Iceberg catalog extend DelegatingCatalogExtension?

We do want to use v2 commands for custom catalogs that do not extend DelegatingCatalogExtension

Does Iceberg catalog extend DelegatingCatalogExtension?

Nope.

We do want to use v2 commands for custom catalogs that do not extend DelegatingCatalogExtension

Even so, is it the right time to introduce such a behavior change in a bug fix release?

We can consider it as a bug. People implementing DS V2 catalog APIs expect to see v2 commands to customize the table behaviors. And there is a backdoor: DelegatingCatalogExtension.

For iceberg, it should be easy to work around it by extending DelegatingCatalogExtension? Iceberg catalog can still keep all its methods unchanged, don't use the delegate.

Iceberg's SparkSessionCatalog already extends a base class. There's no easy way to extend DelegatingCatalogExtension without a major refactoring.

We need to make either the iceberg BaseCatalog or the Spark DelegatingCatalogExtension an interface. It looks easier to make BaseCatalog an interface?

### What changes were proposed in this pull request? This PR updates `DelegatingCatalogExtension` so that it's more extendable - `initialize` becomes not final, so that sub-classes can overwrite it - `delegate` becomes `protected`, so that sub-classes can access it In addition, this PR fixes a mistake that `DelegatingCatalogExtension` is just a convenient default implementation, it's actually the `CatalogExtension` interface that indicates this catalog implementation will delegate requests to the Spark session catalog. #47724 should use `CatalogExtension` instead. ### Why are the changes needed? Unblock the Iceberg extension. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? existing tests ### Was this patch authored or co-authored using generative AI tooling? no Closes #48257 from cloud-fan/catalog. Lead-authored-by: Wenchen Fan <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This PR updates `DelegatingCatalogExtension` so that it's more extendable - `initialize` becomes not final, so that sub-classes can overwrite it - `delegate` becomes `protected`, so that sub-classes can access it In addition, this PR fixes a mistake that `DelegatingCatalogExtension` is just a convenient default implementation, it's actually the `CatalogExtension` interface that indicates this catalog implementation will delegate requests to the Spark session catalog. #47724 should use `CatalogExtension` instead. ### Why are the changes needed? Unblock the Iceberg extension. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? existing tests ### Was this patch authored or co-authored using generative AI tooling? no Closes #48257 from cloud-fan/catalog. Lead-authored-by: Wenchen Fan <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 339dd5b) Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This PR updates `DelegatingCatalogExtension` so that it's more extendable - `initialize` becomes not final, so that sub-classes can overwrite it - `delegate` becomes `protected`, so that sub-classes can access it In addition, this PR fixes a mistake that `DelegatingCatalogExtension` is just a convenient default implementation, it's actually the `CatalogExtension` interface that indicates this catalog implementation will delegate requests to the Spark session catalog. apache/spark#47724 should use `CatalogExtension` instead. ### Why are the changes needed? Unblock the Iceberg extension. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? existing tests ### Was this patch authored or co-authored using generative AI tooling? no Closes #48257 from cloud-fan/catalog. Lead-authored-by: Wenchen Fan <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

github-actions bot added the SQL label Aug 12, 2024

update

09fd19e

cloud-fan reviewed Aug 12, 2024

View reviewed changes

Update sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis…

4553339

…/ResolveSessionCatalog.scala

cloud-fan approved these changes Aug 13, 2024

View reviewed changes

cloud-fan closed this Aug 13, 2024

manuzhang reviewed Sep 19, 2024

View reviewed changes

manuzhang mentioned this pull request Sep 19, 2024

Build: Bump Spark 3.5 to 3.5.3 apache/iceberg#11160

Closed

cloud-fan mentioned this pull request Sep 26, 2024

[SPARK-49791][SQL] Make DelegatingCatalogExtension more extendable #48257

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-49152][SQL] V2SessionCatalog should use V2Command #47724

[SPARK-49152][SQL] V2SessionCatalog should use V2Command #47724

Uh oh!

amaliujia commented Aug 12, 2024 •

edited

Loading

Uh oh!

amaliujia commented Aug 12, 2024

Uh oh!

cloud-fan Aug 12, 2024

Uh oh!

cloud-fan commented Aug 13, 2024

Uh oh!

manuzhang Sep 19, 2024

Uh oh!

cloud-fan Sep 19, 2024

Uh oh!

cloud-fan Sep 19, 2024

Uh oh!

manuzhang Sep 19, 2024 •

edited

Loading

Uh oh!

cloud-fan Sep 19, 2024

Uh oh!

manuzhang Sep 20, 2024

Uh oh!

cloud-fan Sep 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	// For CREATE TABLE [AS SELECT], we should use the v1 command if the catalog is resolved to the
	// session catalog and the table provider is not v2.
	case c @ CreateTable(ResolvedV1Identifier(ident), _, _, tableSpec: TableSpec, _) =>
	val (storageFormat, provider) = getStorageFormatAndProvider(
	c.tableSpec.provider, tableSpec.options, c.tableSpec.location, c.tableSpec.serde,
	ctas = false)
	if (!isV2Provider(provider)) {
	constructV1TableCmd(None, c.tableSpec, ident, c.tableSchema, c.partitioning,
	c.ignoreIfExists, storageFormat, provider)
	} else {
	c
	}

[SPARK-49152][SQL] V2SessionCatalog should use V2Command #47724

[SPARK-49152][SQL] V2SessionCatalog should use V2Command #47724

Uh oh!

Conversation

amaliujia commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

amaliujia commented Aug 12, 2024

Uh oh!

cloud-fan Aug 12, 2024

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Aug 13, 2024

Uh oh!

manuzhang Sep 19, 2024

Choose a reason for hiding this comment

Uh oh!

cloud-fan Sep 19, 2024

Choose a reason for hiding this comment

Uh oh!

cloud-fan Sep 19, 2024

Choose a reason for hiding this comment

Uh oh!

manuzhang Sep 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan Sep 19, 2024

Choose a reason for hiding this comment

Uh oh!

manuzhang Sep 20, 2024

Choose a reason for hiding this comment

Uh oh!

cloud-fan Sep 20, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

amaliujia commented Aug 12, 2024 •

edited

Loading

manuzhang Sep 19, 2024 •

edited

Loading