Skip to content

[SPARK-35194][SQL] Refactor nested column aliasing for readability #32301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
karenfeng wants to merge 22 commits into from
Closed

[SPARK-35194][SQL] Refactor nested column aliasing for readability #32301

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
37ac8a9
Refactor NestedColumnAliasing
karenfeng Apr 21, 2021
9656899
Continue cleanup
karenfeng Apr 22, 2021
e44c683
Add comments
karenfeng Apr 23, 2021
a95360a
Split out Option[Map]
karenfeng Apr 23, 2021
6cfd826
Merge from master
karenfeng Apr 27, 2021
cfa0273
Merge from master
karenfeng Apr 27, 2021
c0f839c
Revert "Merge from master"
karenfeng Apr 27, 2021
a2759cc
Revert "Merge from master"
karenfeng Apr 27, 2021
e48cf77
Merge branch 'master' of https://github.com/apache/spark into refacto…
karenfeng Apr 27, 2021
c4587c1
Remove unused fns
karenfeng Apr 27, 2021
f48b376
Address comments
karenfeng Apr 27, 2021
69ca24e
Indentation
karenfeng Apr 27, 2021
9d41de6
Try to fix compilation
karenfeng Apr 28, 2021
e47a441
mutable.ArrayBuffer toSeq
karenfeng Apr 28, 2021
d74b569
Change alias name
karenfeng Apr 29, 2021
351e74a
Address commen ts
karenfeng May 17, 2021
c7e5c5a
Fixup
karenfeng May 17, 2021
f49663c
Switch away from mapValues
karenfeng May 17, 2021
80fa5c3
Try to fix build
karenfeng May 18, 2021
31f8bd0
Address some comments
karenfeng May 26, 2021
83e2611
Distinct-ify Attributes by exprId
karenfeng May 27, 2021
8a29e94
Only add ExtractValue if 1 reference
karenfeng May 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions ...catalyst/src/main/scala-2.12/org/apache/spark/sql/catalyst/expressions/AttributeMap.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ package org.apache.spark.sql.catalyst.expressions
* of the name, or the expected nullability).
*/
object AttributeMap {
def apply[A](kvs: Map[Attribute, A]): AttributeMap[A] = {
Copy link
Contributor

@cloud-fan cloud-fan May 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of adding a new API, can the caller side just do AttributeMap(map.toSeq)?

Copy link
Contributor Author

@karenfeng karenfeng May 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would work as well, but AttributeMap(kvs = map.toSeq) will end up calling .toMap down the line.

Copy link
Contributor

@cloud-fan cloud-fan May 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah got it, this is more efficient

new AttributeMap(kvs.map(kv => (kv._1.exprId, kv)))
}

def apply[A](kvs: Seq[(Attribute, A)]): AttributeMap[A] = {
new AttributeMap(kvs.map(kv => (kv._1.exprId, kv)).toMap)
}
Expand All @@ -37,6 +41,8 @@ class AttributeMap[A](val baseMap: Map[ExprId, (Attribute, A)])

override def get(k: Attribute): Option[A] = baseMap.get(k.exprId).map(_._2)

override def getOrElse[B1 >: A](k: Attribute, default: => B1): B1 = get(k).getOrElse(default)

override def contains(k: Attribute): Boolean = get(k).isDefined

override def + [B1 >: A](kv: (Attribute, B1)): Map[Attribute, B1] = baseMap.values.toMap + kv
Expand Down
6 changes: 6 additions & 0 deletions ...catalyst/src/main/scala-2.13/org/apache/spark/sql/catalyst/expressions/AttributeMap.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ package org.apache.spark.sql.catalyst.expressions
* of the name, or the expected nullability).
*/
object AttributeMap {
def apply[A](kvs: Map[Attribute, A]): AttributeMap[A] = {
new AttributeMap(kvs.map(kv => (kv._1.exprId, kv)))
}

def apply[A](kvs: Seq[(Attribute, A)]): AttributeMap[A] = {
new AttributeMap(kvs.map(kv => (kv._1.exprId, kv)).toMap)
}
Expand All @@ -37,6 +41,8 @@ class AttributeMap[A](val baseMap: Map[ExprId, (Attribute, A)])

override def get(k: Attribute): Option[A] = baseMap.get(k.exprId).map(_._2)

override def getOrElse[B1 >: A](k: Attribute, default: => B1): B1 = get(k).getOrElse(default)

override def contains(k: Attribute): Boolean = get(k).isDefined

override def updated[B1 >: A](key: Attribute, value: B1): Map[Attribute, B1] =
Expand Down
Loading