Skip to content

[SPARK-29644][SQL] Corrected ShortType and ByteType mapping to SmallInt and TinyInt in JDBCUtils #26301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shivsood wants to merge 5 commits into from
Closed

[SPARK-29644][SQL] Corrected ShortType and ByteType mapping to SmallInt and TinyInt in JDBCUtils #26301

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e13326d
Fixed ShortType wrongly set as Int in JDBCUtils.scala. Added test cas…
shivsood Oct 30, 2019
395fe99
fix for ShortType and ByteType in JDBCUtils.scala, Unit test cases in…
shivsood Nov 1, 2019
66dfd4c
test fixes
shivsood Nov 1, 2019
9cdad2b
fixed test failure in JDBCSuite.scala to get tinyint and smallint as …
shivsood Nov 9, 2019
0600894
removed redundant test cases.
shivsood Nov 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions ...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,4 +202,25 @@ class MsSqlServerIntegrationSuite extends DockerJDBCIntegrationSuite {
df2.write.jdbc(jdbcUrl, "datescopy", new Properties)
df3.write.jdbc(jdbcUrl, "stringscopy", new Properties)
}

test("Write tables with ShortType") {
Copy link
Member

@maropu maropu Nov 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a bug, so can you append the JIRA ID in the prefix?

Copy link
Contributor Author

@shivsood shivsood Nov 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maropu is JIRA addition required at all places i have added a new test cases?

Copy link
Member

@maropu maropu Nov 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, plz follow test("SPARK-29644: test title")

import testImplicits._
val df = Seq(-32768.toShort, 0.toShort, 1.toShort, 38.toShort, 32768.toShort).toDF("a")
val tablename = "shorttable"
df.write
.format("jdbc")
.mode("overwrite")
.option("url", jdbcUrl)
.option("dbtable", tablename)
.save()
val df2 = spark.read
.format("jdbc")
.option("url", jdbcUrl)
.option("dbtable", tablename)
.load()
assert(df.count == df2.count)
val rows = df2.collect()
val colType = rows(0).toSeq.map(x => x.getClass.toString)
assert(colType(0) == "class java.lang.Short")
}
}
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -546,7 +546,7 @@ object JdbcUtils extends Logging {

case ShortType =>
(stmt: PreparedStatement, row: Row, pos: Int) =>
stmt.setInt(pos + 1, row.getShort(pos))
stmt.setShort(pos + 1, row.getShort(pos))

case ByteType =>
(stmt: PreparedStatement, row: Row, pos: Int) =>
Expand Down