[SPARK-34200][SQL] Ambiguous column reference should consider attribute availability

[cloud-fan]

What changes were proposed in this pull request?

This is a long-standing bug that exists since we have the ambiguous self-join check. A column reference is not ambiguous if it can only come from one join side (e.g. the other side has a project to only pick a few columns). An example is

Join(b#1 = 3)
  TableScan(t, [a#0, b#1])
  Project(a#2)
    TableScan(t, [a#2, b#3])

It's a self-join, but b#1 is not ambiguous because it can't come from the right side, which only has column a.

Why are the changes needed?

to not fail valid self-join queries.

Does this PR introduce any user-facing change?

yea as a bug fix

How was this patch tested?

a new test

[cloud-fan]

cc @Ngone51 @viirya @maropu

[Ngone51]

LGTM!

[viirya]

lgtm

[viirya]

BTW, the Join example in the PR description is not aligned properly.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38939/

[cloud-fan]

Ah actually this fixes a 3.1 regression. Previously we merged a bug fix: #30488 , which makes sure ambiguous self-join check is always applied. That said, in 3.0 the ambiguous self-join check is skipped under some cases, which hides the bug this PR is fixing.

For the query below, it works in 3.0, but fails in 3.1. After this PR, it works again.

sql("create table t1 using json as select 1 key, 1 value")
sql("create table t2 using json as select 1 key, 2 value")
val t1 = spark.table("t1")
val t2 = spark.table("t2")
val t3 = t1.join(t2, t1("key") === t2("key")).select(t1("value"))
t1.join(t3, t1("key") > 1)

cc @HyukjinKwon @dongjoon-hyun I think it's a 3.1.1 blocker.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38939/

[SparkQA]

Test build #134353 has finished for PR 31287 at commit 2c34ade.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master, branch-3.1 and branch-3.0.