fix: Fall back to Spark for MakeDecimal with unsupported input type #2815
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2815 +/- ##
============================================
+ Coverage 56.12% 59.14% +3.02%
- Complexity 976 1471 +495
============================================
Files 119 167 +48
Lines 11743 15238 +3495
Branches 2251 2514 +263
============================================
+ Hits 6591 9013 +2422
- Misses 4012 4942 +930
- Partials 1140 1283 +143 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| } | ||
| let result_type = DataType::Decimal128(precision, scale); | ||
| ColumnarValue::Array(a) => match a.data_type() { | ||
| DataType::Int64 => { |
There was a problem hiding this comment.
should we also support Int32? 🤔
There was a problem hiding this comment.
for this PR, I just wanted to fall back to avoid crashing. We could choose to add support in the future though
| SQLConf.ANSI_ENABLED.key -> "false", | ||
| SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false", | ||
| CometConf.getExprAllowIncompatConfigKey(classOf[Sum]) -> "true", | ||
| CometConf.COMET_NATIVE_SCAN_IMPL.key -> CometConf.SCAN_NATIVE_ICEBERG_COMPAT, |
There was a problem hiding this comment.
Are all settings needed for this test ? E.g. Sum and Iceberg compat look unrelated ?
spark/src/main/scala/org/apache/comet/serde/decimalExpressions.scala
Outdated
Show resolved
Hide resolved
| checkSparkAnswerAndFallbackReason(df1, "Unsupported input data type: IntegerType") | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
IMO it would be useful to add a positive test too - with LongType
….scala Co-authored-by: Martin Grigorov <[email protected]>
|
are we waiting to address any feedback on this PR? |
4 participants
Which issue does this PR close?
Closes #2813
Rationale for this change
What changes are included in this PR?
How are these changes tested?