feat: use Substrait's PrecisionTimestamp and PrecisionTimestampTz instead of deprecated Timestamp #11597
Conversation
Blizzara
force-pushed
the
avo/precision-timestamp
branch
from
49d90c8 to
350e00f
Compare
Blizzara
force-pushed
the
avo/precision-timestamp
branch
from
350e00f to
e8753f6
Compare
| r#type::Kind::IntervalYear(_) => { | ||
| Ok(DataType::Interval(IntervalUnit::YearMonth)) | ||
| } | ||
| r#type::Kind::IntervalDay(_) => Ok(DataType::Interval(IntervalUnit::DayTime)), |
There was a problem hiding this comment.
this was just cleanup - we don't check type variations for types where we don't use them
| })) => { | ||
| // DF only supports millisecond precision, so we lose the micros here | ||
| ScalarValue::new_interval_dt(*days, (seconds * 1000) + (microseconds / 1000)) | ||
| // DF only supports millisecond precision, so for any more granular type we lose precision |
There was a problem hiding this comment.
these changes were needed as part of bumping substrait (substrait-io/substrait#665)
| kind: Some(r#type::Kind::IntervalDay(r#type::IntervalDay { | ||
| type_variation_reference: DEFAULT_TYPE_VARIATION_REF, | ||
| nullability, | ||
| precision: Some(3), // DayTime precision is always milliseconds |
There was a problem hiding this comment.
required due to bumping substrait substrait-io/substrait#665
Blizzara
force-pushed
the
avo/precision-timestamp
branch
from
e8753f6 to
a639feb
Compare
| substrait = { version = "0.36.0", features = ["serde"] } | ||
| pbjson-types = "0.7" | ||
| prost = "0.13" | ||
| substrait = { version = "0.41", features = ["serde"] } |
There was a problem hiding this comment.
need a bump for precision timestamp types to have precision, and their value as i64 instead of u64
pbjson and prost need to be bumped to match substrait's deps
| join_rel::JoinType::Anti => Ok(JoinType::LeftAnti), | ||
| join_rel::JoinType::Semi => Ok(JoinType::LeftSemi), | ||
| join_rel::JoinType::LeftAnti => Ok(JoinType::LeftAnti), | ||
| join_rel::JoinType::LeftSemi => Ok(JoinType::LeftSemi), |
There was a problem hiding this comment.
needed due to bumping substrait - I think this is just a compile-time break tho, the actual protobuf values stay the same
Blizzara
force-pushed
the
avo/precision-timestamp
branch
from
a639feb to
5347f30
Compare
| substrait = { version = "0.36.0", features = ["serde"] } | ||
| pbjson-types = "0.7" | ||
| prost = "0.13" | ||
| substrait = { version = "0.41", features = ["serde"] } |
| pub const DEFAULT_TYPE_VARIATION_REF: u32 = 0; | ||
| pub const UNSIGNED_INTEGER_TYPE_VARIATION_REF: u32 = 1; | ||
|
|
||
| #[deprecated(since = "41.0.0", note = "Use `PrecisionTimestamp(Tz)` type instead")] |
There was a problem hiding this comment.
SInce we have already released 41 https://crates.io/crates/datafusion/41.0.0 I think we should probably label this with the next version. For example:
| #[deprecated(since = "41.0.0", note = "Use `PrecisionTimestamp(Tz)` type instead")] | |
| #[deprecated(since = "42.0.0", note = "Use `PrecisionTimestamp(Tz)` type instead")] |
There was a problem hiding this comment.
Hah yeah, this PR was a long time in the making 😅 fixed in a05637b
|
Thanks again @Blizzara |
2 participants
Which issue does this PR close?
N/A
closes #12074
Rationale for this change
DF was using the Substrait type_variations on a Substrait Timestamp to indicate whether a timestamp is seconds/millis/micros/nanos, while that works within DF that's not interoperable with other systems. Substrait has since introduced a PrecisionTimestamp type which includes the precision as a first class option, so we should use that instead.
What changes are included in this PR?
Are these changes tested?
Yes, with unit tests
Are there any user-facing changes?