Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode #36667
Conversation
Summary of ChangesHello @chenxuesdu, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the Spanner change stream connector by introducing logic to adapt query behavior for PostgreSQL dialects based on the change stream's partition mode. Specifically, it allows the system to correctly query change streams that use either MUTABLE_KEY_RANGE or IMMUTABLE_KEY_RANGE partitioning by selecting the appropriate internal Spanner Table-Valued Function, ensuring proper data retrieval and compatibility. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
chenxuesdu
changed the title
chenxuesdu
force-pushed
the
master
branch
8 times, most recently
from
473321f to
88ea37b
Compare
|
Assigning reviewers: R: @m-trieu for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| Statement.newBuilder( | ||
| "select option_name, option_value\n" | ||
| + "from information_schema.change_stream_options\n" | ||
| + "where change_stream_name = $1") |
There was a problem hiding this comment.
Can we add option_name = 'partition_mode' here so we can get the parititon mode directly and no need to filter later?
| Statement.newBuilder( | ||
| "select option_name, option_value\n" | ||
| + "from information_schema.change_stream_options\n" | ||
| + "where change_stream_name = @changeStreamName") |
| query = | ||
| "SELECT * FROM \"spanner\".\"read_json_" + changeStreamName + "\"($1, $2, $3, $4, null)"; | ||
| // Ensure we have determined whether change stream uses mutable key range | ||
| boolean isMutable = isMutableKeyRangeChangeStream(); |
There was a problem hiding this comment.
Is this called each time we issue the query? As our query will be constantly checkpointed and canceled and re-issued, this can be called many times.
I think we just need to query the partition mode once for the whole pipeline.
chenxuesdu
force-pushed
the
master
branch
4 times, most recently
from
5d1163c to
d5f8b77
Compare
the change stream partition mode For MUTABLE_KEY_RANGE change stream, use read_proto_bytes_, else use read_json_
2 participants
Add a private variable isMutableKeyRange, which will be initialized during ChangeStreamDao constructor by reading from internal spanner table.
In function changeStreamQuery, when constructing change stream query, based on the value of isMutableKeyRange, call different TVFs when the Dialect is PostgreSQL(The googleSQL part is completed).