[SPARK-47496][SQL] Java SPI Support for dynamic JDBC dialect registering #45626
Conversation
| override def supportsOffset: Boolean = true | ||
| } | ||
|
|
||
| private[jdbc] object OracleDialect { |
There was a problem hiding this comment.
Ur, actually, case object OracleDialect has more features (serializable, hashCode, toString) than object OracleDialect. Do we need to shrink features like this?
There was a problem hiding this comment.
This is only a place for containing some const ints, which represent some variants from Oracle JDBC extensions. Thus, object is enough
There was a problem hiding this comment.
Opps, the errors occured
- simple scan with LIMIT *** FAILED *** (65 milliseconds)
[info] org.apache.spark.SparkException: Task not serializable
| assert(namedObservation.get === expected) | ||
| } | ||
|
|
||
|
|
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM (with minor comments). It's a nice feature, @yaooqinn !
|
Thank you @dongjoon-hyun. I addressed the comments |
|
#45626 (comment) seems to be missed still. Are you going to recover it, @yaooqinn ?
|
|
Yes @dongjoon-hyun it was caused by the unserializable object |
|
@yaooqinn It's really worth? as I know, users could register dialects with |
|
@beliefer Yes, it's worth it. Actually, there are plenty of hacky ways to register dialects just like what you mentioned. We can't anticipate those as typical API usages. |
|
Merged to master. Thank you @dongjoon-hyun |
|
In which version will it be released, and can multiple libraries be joined for querying? |
|
@yaooqinn is it only in Spark 4 or 3 as well? |
|
@elijah-pl please refer to the JIRA sizde for fixed versions |
5 participants
What changes were proposed in this pull request?
This PR brings the Java ServiceProvider Interface (SPI) Support for dynamic JDBC dialect registering.
A custom JDBC dialect can be registered easily instead of calling JdbcDialects.registerDialect manually.
Why are the changes needed?
For pure SQL and other non-Java API users, it's difficult to register a custom JDBC dialect to use. With this patch, this can be done when the jar containing the dialect class is visible to the spark classloader.
Does this PR introduce any user-facing change?
Yes, but mostly for third-party developers
How was this patch tested?
new tests
Was this patch authored or co-authored using generative AI tooling?
no