feat: Support Binary bitwise shift operators (<< and >>) #3037

ovr · 2022-08-04T17:21:18Z

Which issue does this PR close?

This PR introduces support for binary bitwise shift operators like >> and <<.

Refs #1619

Are there any user-facing changes?

It's new functionality, and there are no breaking changes.

Thanks!

ovr · 2022-08-04T18:17:46Z

Right now, DF uses Generic Dialect for sqlparser, but it's allowed only for PostgreSqlDialect

Token::ShiftLeft if dialect_of!(self is PostgreSqlDialect) => {
    Some(BinaryOperator::PGBitwiseShiftLeft)
}
Token::ShiftRight if dialect_of!(self is PostgreSqlDialect) => {
    Some(BinaryOperator::PGBitwiseShiftRight)
}

@alamb what should I do with it? Should I Introduce new methods for Dialect to allow these operators with Generic dialect?

Thanks

alamb · 2022-08-04T18:43:19Z

@alamb what should I do with it? Should I Introduce new methods for Dialect to allow these operators with Generic dialect?

What about allowing BinaryOperator::PGBitwiseShiftRight in GenericDialect ? I think the GenericDialect in sqlparser is sort of a "anything goes" dialect so it should be fine to extend it to allow these operators.

I can make another sql-parser release soon if you need one as well

alamb

I looked at the PR and I think the basic direction is very nice 👍 thanks @ovr

alamb · 2022-08-04T18:43:50Z

datafusion/physical-expr/src/expressions/binary/kernels.rs

    }
 }

+pub(crate) fn bitwise_shift_right(left: ArrayRef, right: ArrayRef) -> Result<ArrayRef> {


liukun4515 · 2022-08-05T09:37:53Z

@alamb what should I do with it? Should I Introduce new methods for Dialect to allow these operators with Generic dialect?

What about allowing BinaryOperator::PGBitwiseShiftRight in GenericDialect ? I think the GenericDialect in sqlparser is sort of a "anything goes" dialect so it should be fine to extend it to allow these operators.

I can make another sql-parser release soon if you need one as well

LGTM, if we can merge the sql-rs first

ovr · 2022-08-05T14:43:44Z

Looks likes I forget about overflowing here. I think I can use overflowing_shl to handle it correctly to the Error https://doc.rust-lang.org/std/primitive.i64.html#method.overflowing_shl

❯ select 2 << 1024;
thread 'main' panicked at 'attempt to shift left with overflow', datafusion/physical-expr/src/expressions/binary.rs:524:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

ovr · 2022-08-05T15:12:23Z

Panic on overflow was fixed in 474ed5a

Without error to be similar with PostgreSQL

alamb · 2022-08-06T14:36:21Z

https://crates.io/crates/sqlparser/0.20.0 is released, FYI

ovr · 2022-08-08T11:11:13Z

Awaiting #3072

alamb · 2022-08-08T21:25:52Z

Awaiting #3072

Is merged 🎉

ovr · 2022-08-11T20:59:07Z

datafusion/physical-expr/src/expressions/binary/kernels.rs

 /// The binary_bitwise_array_op macro only evaluates for integer types
 /// like int64, int32.
 /// It is used to do bitwise operation on an array with a scalar.
 macro_rules! binary_bitwise_array_scalar {


I've changed binary_bitwise_array_scalar to pass function as expr to do casting, because wrapping_shr and wrapping_shl requires u32. I tried to do it with let right: $TYPE = scalar.try_into().unwrap(); but I found that it will show a error because it's not possible to cast all numeric types to u32.

I think it makes sense -- thank you

codecov-commenter · 2022-08-11T21:51:42Z

Codecov Report

Merging #3037 (810f4b0) into master (48f9b7a) will decrease coverage by 0.03%.
The diff coverage is 63.15%.

@@            Coverage Diff             @@
##           master    #3037      +/-   ##
==========================================
- Coverage   85.95%   85.91%   -0.04%     
==========================================
  Files         291      291              
  Lines       52382    52475      +93     
==========================================
+ Hits        45025    45084      +59     
- Misses       7357     7391      +34

Impacted Files	Coverage Δ
...on/physical-expr/src/expressions/binary/kernels.rs	`52.17% <44.44%> (-10.99%)`	⬇️
datafusion/physical-expr/src/expressions/binary.rs	`97.71% <93.10%> (-0.17%)`	⬇️
datafusion/core/tests/sql/expr.rs	`99.84% <100.00%> (+<0.01%)`	⬆️
datafusion/expr/src/binary_rule.rs	`84.61% <100.00%> (ø)`
datafusion/expr/src/operator.rs	`96.00% <100.00%> (+0.16%)`	⬆️
datafusion/sql/src/planner.rs	`81.96% <100.00%> (+0.01%)`	⬆️
datafusion/core/src/physical_plan/metrics/value.rs	`86.93% <0.00%> (-0.51%)`	⬇️
datafusion/expr/src/logical_plan/plan.rs	`77.60% <0.00%> (-0.18%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

alamb

This PR is looking pretty good 👍

Please mark it as "ready for review" when it is ready.

Thanks !

ovr · 2022-08-12T12:32:58Z

Marked PR as ready for review ✅

alamb

Looks great to me @ovr - thank you for the contribution!

I had one tiny testing suggestion, but otherwise I think this PR is ready to go

alamb · 2022-08-12T20:27:15Z

datafusion/physical-expr/src/expressions/binary.rs

+        let modules = Arc::new(Int32Array::from(vec![Some(100)])) as ArrayRef;
+        let result = bitwise_shift_left(input.clone(), modules.clone())?;
+
+        let expected = Int32Array::from(vec![Some(32)]);


TIL 2 << 100 = 32 🤯

https://play.rust-lang.org/?version=stable&mode=release&edition=2018&gist=b0682ca51f798793e784b852e28dc4a8

2<<100 =>> 2<<(100%bit_width(i32)) =>> 2 << 4?

i am surprised that this is rotational shifting

i am surprised that this is rotational shifting

me too...

alamb · 2022-08-12T20:28:21Z

datafusion/physical-expr/src/expressions/binary.rs

+    #[test]
+    fn bitwise_shift_array_test() -> Result<()> {
+        let input = Arc::new(Int32Array::from(vec![Some(2), None, Some(10)])) as ArrayRef;
+        let modules =


I suggest a test for when the modules is Null (you cover NULL for the input already)

alamb · 2022-08-12T20:28:35Z

datafusion/physical-expr/src/expressions/binary.rs

+    #[test]
+    fn bitwise_shift_scalar_test() -> Result<()> {
+        let input = Arc::new(Int32Array::from(vec![Some(2), None, Some(4)])) as ArrayRef;
+        let module = ScalarValue::from(10i32);


Likewise, here a test for null handling might be good

Instead of doing unit test here, I did a simple SQL test which test the whole pipeline, and I found that DF doesn't handle nulls for bitwise operators.

Fixed in:

810f4b0

alamb · 2022-08-12T20:29:06Z

datafusion/physical-expr/src/expressions/binary/kernels.rs

 /// The binary_bitwise_array_op macro only evaluates for integer types
 /// like int64, int32.
 /// It is used to do bitwise operation on an array with a scalar.
 macro_rules! binary_bitwise_array_scalar {


I think it makes sense -- thank you

ovr · 2022-08-14T22:12:01Z

datafusion/expr/src/binary_rule.rs

    use arrow::datatypes::DataType::*;

-    if !is_numeric(left_type) || !is_numeric(right_type) {
+    if !both_numeric_or_null_and_numeric(left_type, right_type) {


To be similar with default SQL behaviour for null.

datafusion/expr/src/operator.rs

Co-authored-by: Jiayu Liu <[email protected]>

andygrove · 2022-08-15T14:35:44Z

Can we update the user guide to show this new functionality? This can be a follow-on issue.

ovr · 2022-08-15T14:43:06Z

@andygrove

Can we update the user guide to show this new functionality? This can be a follow-on issue.

https://github.com/apache/arrow-datafusion/blob/master/docs/source/user-guide/sql/sql_status.md#supported-sql

I didn't find any place where I should update the documentation, as I can see there is a small comment about binary expressions, but it doesn't include all binary expressions.

Many mathematical unary and binary expressions such as +, /, sqrt, tan, >=.

I updated the description on this PR to reference #1619

alamb

Thanks again @ovr

ursabot · 2022-08-15T17:01:58Z

Benchmark runs are scheduled for baseline = 5ddad47 and contender = ee55d89. ee55d89 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

github-actions bot added core Core DataFusion crate logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sql SQL Planner labels Aug 4, 2022

ovr force-pushed the binary-bitwise-shifts-pr branch 2 times, most recently from a5a06ff to 6449570 Compare August 4, 2022 17:29

alamb reviewed Aug 4, 2022

View reviewed changes

alamb mentioned this pull request Aug 4, 2022

[postgres] Add support for custom binary operators apache/datafusion-sqlparser-rs#548

Merged

ovr mentioned this pull request Aug 4, 2022

feat: Allow to use >> and << binary operators in Generic dialect apache/datafusion-sqlparser-rs#553

Merged

ovr force-pushed the binary-bitwise-shifts-pr branch from 4e1dfde to 36aff67 Compare August 5, 2022 14:46

waitingkuo mentioned this pull request Aug 8, 2022

Update sql parser to v0.20.0 #3072

Merged

ovr force-pushed the binary-bitwise-shifts-pr branch from 474ed5a to 636be02 Compare August 11, 2022 20:56

ovr commented Aug 11, 2022

View reviewed changes

ovr marked this pull request as ready for review August 11, 2022 20:59

ovr force-pushed the binary-bitwise-shifts-pr branch from 636be02 to 17fb689 Compare August 11, 2022 21:06

ovr marked this pull request as draft August 11, 2022 21:14

ovr force-pushed the binary-bitwise-shifts-pr branch from 17fb689 to 92fa53c Compare August 11, 2022 21:17

alamb reviewed Aug 12, 2022

View reviewed changes

ovr marked this pull request as ready for review August 12, 2022 12:32

alamb approved these changes Aug 12, 2022

View reviewed changes

ovr added 2 commits August 15, 2022 01:00

feat: Support Binary bitwise shift operators (<< and >>)

5628e74

no panic on overflow

624e8b3

ovr force-pushed the binary-bitwise-shifts-pr branch from 92fa53c to 624e8b3 Compare August 14, 2022 22:01

null coercion test + fix

810f4b0

ovr commented Aug 14, 2022

View reviewed changes

jimexist reviewed Aug 15, 2022

View reviewed changes

datafusion/expr/src/operator.rs Outdated Show resolved Hide resolved

jimexist approved these changes Aug 15, 2022

View reviewed changes

Update datafusion/expr/src/operator.rs

fb73fc1

Co-authored-by: Jiayu Liu <[email protected]>

alamb approved these changes Aug 15, 2022

View reviewed changes

alamb merged commit ee55d89 into apache:master Aug 15, 2022

ovr deleted the binary-bitwise-shifts-pr branch August 16, 2022 08:39

isidentical mentioned this pull request Aug 27, 2022

Fix propagation of optimized predicates on nested projections #3228

Merged

feat: Support Binary bitwise shift operators (<< and >>) #3037

feat: Support Binary bitwise shift operators (<< and >>) #3037

Uh oh!

Conversation

ovr commented Aug 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Are there any user-facing changes?

Uh oh!

ovr commented Aug 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Aug 4, 2022

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liukun4515 commented Aug 5, 2022

Uh oh!

ovr commented Aug 5, 2022

Uh oh!

ovr commented Aug 5, 2022

Uh oh!

alamb commented Aug 6, 2022

Uh oh!

ovr commented Aug 8, 2022

Uh oh!

alamb commented Aug 8, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Aug 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

ovr commented Aug 12, 2022

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liukun4515 Aug 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ovr Aug 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andygrove commented Aug 15, 2022

Uh oh!

ovr commented Aug 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

ursabot commented Aug 15, 2022

Uh oh!

ovr commented Aug 4, 2022 •

edited

Loading

ovr commented Aug 4, 2022 •

edited

Loading

codecov-commenter commented Aug 11, 2022 •

edited

Loading

liukun4515 Aug 13, 2022 •

edited

Loading

ovr Aug 14, 2022 •

edited

Loading

ovr commented Aug 15, 2022 •

edited

Loading