A query to redshift (example:
val res = queryFunc("select count(column.name) as notnull from features_permanent") )
may return either a BigInt (resolving as Long) or Double (resolving as Double).
This may derive from code copied in from spark in commit 2cd12e7, and seems to derive from apache/spark@a1e092e, which assigns type based on whether the result is signed.
While this may be a proper result for generic jdbc connected database bigints (arguable), redshift does not have unsigned bigints, and redshift bigints should be resolved to longs.
Workaround are possible based on detecting type at runtime, but are ugly.
A pull request fixing https://github.com/databricks/spark-redshift/blob/master/src/main/scala/com/databricks/spark/redshift/RedshiftJDBCWrapper.scala#L304 will follow shortly.
A query to redshift (example:
val res = queryFunc("select count(column.name) as notnull from features_permanent"))may return either a BigInt (resolving as Long) or Double (resolving as Double).
This may derive from code copied in from spark in commit 2cd12e7, and seems to derive from apache/spark@a1e092e, which assigns type based on whether the result is signed.
While this may be a proper result for generic jdbc connected database bigints (arguable), redshift does not have unsigned bigints, and redshift bigints should be resolved to longs.
Workaround are possible based on detecting type at runtime, but are ugly.
A pull request fixing https://github.com/databricks/spark-redshift/blob/master/src/main/scala/com/databricks/spark/redshift/RedshiftJDBCWrapper.scala#L304 will follow shortly.