Skip to content

[SPARK-54240] Translate get array item catalyst expression to connector expression #52940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
+84 −1
Open

[SPARK-54240] Translate get array item catalyst expression to connector expression #52940

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
276fbc5
Translate get array item catalyst expression to connector expression
urosstan-db Nov 7, 2025
211e8d2
Implement ToStringSQLBuilder
urosstan-db Nov 7, 2025
7b29373
Use separate expression for get array item translation
urosstan-db Nov 10, 2025
395361b
Introduce new error code
urosstan-db Nov 10, 2025
ebc3dd1
Fix building of get array item
urosstan-db Nov 10, 2025
bbdd90a
Rename getters
urosstan-db Nov 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions sql/catalyst/src/main/java/org/apache/spark/sql/connector/util/V2ExpressionSQLBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ yield visitBinaryArithmetic(
case "LTRIM" -> visitTrim("LEADING", expressionsToStringArray(e.children()));
case "RTRIM" -> visitTrim("TRAILING", expressionsToStringArray(e.children()));
case "OVERLAY" -> visitOverlay(expressionsToStringArray(e.children()));
case "GET_ARRAY_ITEM" -> visitGetArrayItem(expressionsToStringArray(e.children()));
// TODO supports other expressions
default -> visitUnexpectedExpr(expr);
};
Expand Down Expand Up @@ -342,6 +343,13 @@ protected String visitTrim(String direction, String[] inputs) {
}
}

protected String visitGetArrayItem(String[] inputs) {
throw new SparkUnsupportedOperationException(
"_LEGACY_ERROR_TEMP_3177",
Map.of("class", this.getClass().getSimpleName(), "funcName", "GET_ARRAY_ITEM")
);
}

protected String visitExtract(Extract extract) {
return visitExtract(extract.field(), build(extract.source()));
}
Expand Down
3 changes: 3 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,9 @@ class V2ExpressionBuilder(e: Expression, isPredicate: Boolean = false) extends L
case _: Sha2 => generateExpressionWithName("SHA2", expr, isPredicate)
case _: StringLPad => generateExpressionWithName("LPAD", expr, isPredicate)
case _: StringRPad => generateExpressionWithName("RPAD", expr, isPredicate)
case GetArrayItem(_, _, failOnError) if failOnError =>
// Pushdown only if ANSI is enabled (fail on error) to be compatible with remote systems.
generateExpressionWithName("GET_ARRAY_ITEM", expr, isPredicate)
Copy link
Contributor Author

@urosstan-db urosstan-db Nov 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make new v2 expression as well to be able to pass failOnError argument, since get method in spark returns 0-indexed element of array, but it does not fail for index out of bounds, so it would be beneficial to pass failOnError to be able to support get method pushdown beside bracket access. WDYT @srielau @cloud-fan ?

// TODO supports other expressions
case ApplyFunctionExpression(function, children) =>
val childrenExpressions = children.flatMap(generateExpression(_))
Expand Down