[Feature] add Trino JDBC and\or Arrow Flight SQL JDBC or ADBC 🚀

[YuriyGavrilov]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

First thing is to add existing Trino JDBC Drives Support as source and sink connector.

• Add Trino JDBC as source connector
• Add Trino JDBC as sink connector
• Think about Trino as engine (maybe)
• Add Arrow Flight SQL as source connector ( as JDBC or ADBC ) @sunxiaojian
• Add Arrow Flight SQL as sink connector ( as JDBC or ADBC ) @sunxiaojian

Just to leave this here:

• https://www.vldb.org/pvldb/vol10/p1022-muehleisen.pdf

Some good example of Flight SQL server as reference:

• https://github.com/voltrondata/flight-sql-server-example
• https://www.kamu.dev/blog/2023-09-datafusion-flightsql/ https://github.com/kamu-data/kamu-cli
• https://www.postgresql.org/about/news/apache-arrow-flight-sql-adapter-for-postgresql-010-2716/

Usage Scenario

• Data synchronization from Trino
• Data synchronization to Trino
• Using Trino as SQL engine for transformer ( including data enrichment from source to sink by existing data in Trino )
• Fast data synchronization by Arrow Flight support

Related issues

About Trino Dialect - #4162

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[sanjaychitransh]

hi ,I have completed trino connector development, tested working
I will submit PR

[YuriyGavrilov]

@sanjaychitransh great job thx!

[sunxiaojian]

Is there a real need to synchronize Trino data ? Trino itself can connect everything.

[YuriyGavrilov]

Good question @sunxiaojian
For me there could be cases take data from Trino to somewhere outside.

[sunxiaojian]

Good question @sunxiaojian For me there could be cases take data from Trino to somewhere outside.

I understand that Trino itself does not store data; instead, its data sources are the connector components it supports. Synchronizing Trino data essentially involves synchronizing the data from these underlying plugins. Does this mean we can bypass Trino and directly synchronize data from the underlying storage systems?

[YuriyGavrilov]

Good question @sunxiaojian For me there could be cases take data from Trino to somewhere outside.

I understand that Trino itself does not store data; instead, its data sources are the connector components it supports. Synchronizing Trino data essentially involves synchronizing the data from these underlying plugins. Does this mean we can bypass Trino and directly synchronize data from the underlying storage systems?

Hi @sunxiaojian Trino could join different source and make some views so it could be complicated to find exactly source attributes in one seatunnel source.

Additionally I make some tests with iceberg table and it is working solution with bypass strategy to sink data to Trino iceberg.

[sunxiaojian]

Good question @sunxiaojian For me there could be cases take data from Trino to somewhere outside.

I understand that Trino itself does not store data; instead, its data sources are the connector components it supports. Synchronizing Trino data essentially involves synchronizing the data from these underlying plugins. Does this mean we can bypass Trino and directly synchronize data from the underlying storage systems?

Hi @sunxiaojian Trino could join different source and make some views so it could be complicated to find exactly source attributes in one seatunnel source.

Additionally I make some tests with iceberg table and it is working solution with bypass strategy to sink data to Trino iceberg.

ok, I understand. Combining data from different sources is a great scenario.

[sunxiaojian]

Let me try to support arrow flight sql. arrow flight sql

[YuriyGavrilov]

Let me try to support arrow flight sql. arrow flight sql

@sunxiaojian wow it is great news, I'm happy to know that there are people who care about arrow :) thx it will be super useful maybe not today but "tomorrow" for sure.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

[YuriyGavrilov]

It is actual