[SPARK-26946][SQL] Identifiers for multi-catalog

[jzhuge]

What changes were proposed in this pull request?

• Support N-part identifier in SQL
• N-part identifier extractor in Analyzer

How was this patch tested?

• A new unit test suite ResolveMultipartRelationSuite
• CatalogLoadingSuite

@rBlue @cloud-fan @mccheah

[dongjoon-hyun]

ok to test. Thank you, @jzhuge !

[jzhuge]

Looking at the build failure

[cloud-fan]

This is a top-level parser entry used in ParserInterface. I don't think we need it now for catalog identifier.

[jzhuge]

True, only my test case uses it to parse a table name into a sequence. I will remove it.

[rdblue]

Won't we need this eventually for parsing names passed to saveAsTable? Why not add it now?

[jzhuge]

When I start to to convert SELECT, INSERT, and DROP code path to support multi-catalog, this parse function is needed, e.g,

  override def visitTable(ctx: TableContext): LogicalPlan = withOrigin(ctx) {
    UnresolvedIdentifier(visitMultiPartIdentifier(ctx.multiPartIdentifier))
  }

 override def visitTableName(ctx: TableNameContext): LogicalPlan = withOrigin(ctx) {
    val tableId = visitMultiPartIdentifier(ctx.multiPartIdentifier())
    val table = mayApplyAliasPlan(ctx.tableAlias, UnresolvedIdentifier(tableId))
    table.optionalMap(ctx.sample)(withSample)
  }

[mccheah]

Sorry for breaking up my review into individual comments. I think this looks ok short of some style changes.

[rdblue]

This uses a lot of temporary classes to simulate future rules that match multi-part identifiers. I think I would rather include an update that adds new UnresolvedRelation nodes and uses them instead of test plan nodes, but I'd be interested to hear whether @cloud-fan agrees.

[jzhuge]

OK either way. I have already converted SELECT/INSERT/DROP code paths to support multi-catalog in my private 2.3 branch. Pretty straightforward. Converting CREATE would be a lot easier with Ryan's PR 24029.

[rdblue]

@jzhuge, this looks really close to being ready to me!

[dilipbiswal]

retest this please

[rdblue]

Minor: Looks like Scaladoc conventions used in Javadoc. This should be {@link Identifier}.

[rdblue]

+1

This looks good to me. @cloud-fan, do you have any more review comments?

[cloud-fan]

Shall we use a class directly? I don't see much value of using an interface here, as it has only one implementation.

[rdblue]

This allows us more flexibility than a single concrete class. Changing a class to an interface is not a binary compatible change, so using an interface is the right thing to do.

[cloud-fan]

Then I suggest we move the impl class to a private package like org.apache.spark.sql.catalyst. Also the static method should be moved to the impl class as well, as we only create it inside Spark.

[rdblue]

The implementation class is package-private. If we were to move it to a different package, we would need to make it public for the of factory method, which would increase its visibility, not decrease it.

[cloud-fan]

Why it's a trait?

My understanding is this PR adds the class of the catalog object identifier and the related parser support. I don't think we have a detailed design of how analyzer looks up catalog yet.

[rdblue]

This trait provides extractors, similar to a trait like PredicateHelper. These implement the resolution rules from the SPIP using a generic catalog lookup provided by the implementation.

This decouples the resolution rules from how the analyzer looks up catalogs and provides convenient extractors that implement those rules.

[cloud-fan]

then this should be an internal trait under a private package like org.apache.spark.sql.catalyst

[SparkQA]

Test build #103750 has finished for PR 23848 at commit 3bb4485.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mccheah]

@cloud-fan have we addressed all your comments, or did you have any other feedback you wanted to give? Would like to merge this soon to unblock other V2 work, particularly table catalogs.

Possibly try to merge before EOD Pacific time today, at the very latest before end of week?

For everyone else following, please feel free to leave any feedback we would like to address before this goes in.

[cloud-fan]

who will call this constructor? I feel we are adding too much code for future use only. Can we add them when they are needed? It will be good if this PR only add the identifier interface and impl class, and the related parser rules, which is pretty easy to justify.

[rdblue]

@cloud-fan, I think this commit is reasonably self-contained. Nit-picking about whether a constructor is added in this commit or the next isn't adding much value.

Keep in mind that we make commits self-contained to decrease conflicts and increase the rate at which we can review and accept patches. Is putting this in the next commit really worth the time it takes to change and test that change, if it means that this work is delayed another day?

[cloud-fan]

The parser part and identifier interface/impl class LGTM. The catalog lookup part looks reasonable but I'm not very confident without seeing the actual use case. To move things forward, I'm merging this. I may refactor this part after the table catalog gets it.

[cloud-fan]

thanks, merging to master!

[jzhuge]

Thanks @cloud-fan !

[rdblue]

Thanks for merging, @cloud-fan, and thanks for working on this, @jzhuge!