[SPARK-28685][SQL][TEST] Test HMS 2.0.0+ in VersionsSuite/HiveClientSuites on JDK 11

[wangyum]

What changes were proposed in this pull request?

It seems Datanucleus 3.x can not support JDK 11:

[info]   Cause: org.datanucleus.exceptions.NucleusException: The java type java.lang.Long (jdbc-type="", sql-type="") cant be mapped for this datastore. No mapping is available.
[info]   at org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.getDatastoreMappingClass(RDBMSMappingManager.java:1215)
[info]   at org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.createDatastoreMapping(RDBMSMappingManager.java:1378)
[info]   at org.datanucleus.store.rdbms.table.AbstractClassTable.addDatastoreId(AbstractClassTable.java:392)
[info]   at org.datanucleus.store.rdbms.table.ClassTable.initializePK(ClassTable.java:1087)
[info]   at org.datanucleus.store.rdbms.table.ClassTable.preInitialize(ClassTable.java:247)

Hive upgrade Datanucleus to 4.x from Hive 2.0(HIVE-6113). This PR makes it skip 0.12, 0.13, 0.14, 1.0, 1.1 and 1.2 when testing with JDK 11.

Note that, this pr will not fix sql read hive materialized view. It's another issue:

3.0: sql read hive materialized view *** FAILED *** (1 second, 521 milliseconds)
3.1: sql read hive materialized view *** FAILED *** (1 second, 536 milliseconds)

How was this patch tested?

manual tests:

export JAVA_HOME="/usr/lib/jdk-11.0.3"
build/sbt "hive/test-only *.VersionsSuite *.HiveClientSuites" -Phive -Phadoop-3.2

[wangyum]

Note that, this pr will not fix sql read hive materialized view. It's another issue:

3.0: sql read hive materialized view *** FAILED *** (1 second, 521 milliseconds)
3.1: sql read hive materialized view *** FAILED *** (1 second, 536 milliseconds)

[SparkQA]

Test build #108919 has finished for PR 25405 at commit fae79ff.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

@wangyum . As you mentioned here, this PR doesn't make them pass on JDK11.
In that case, we had better focus what this PR (AS-IS) did. Let me update your PR title.

[wangyum]

Thank you @dongjoon-hyun

[dongjoon-hyun]

Thank YOU, @wangyum !

+1, LGTM. Merged to master.
(This passed Jenkins with hadoop-2.7 already and the hadoop-3.2 Jenkins run also passed all Java tests).

[SparkQA]

Test build #108925 has finished for PR 25405 at commit fae79ff.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

cc @srowen , @gatorsmile , @cloud-fan since this PR removed the test coverage for old HMS at JDK11. If there is a way to support JDK11 with old HMS libraries, we can re-enable this later.

[gatorsmile]

That sounds good to me.

[wangyum]

@shaneknapp Could we support adding test-java11 signal to the PR title for testing with Java 11?

[HyukjinKwon]

@shaneknapp .. actually, adding test-java11 in PR builder turned to be a bit important now.. @wangyum, mind filing an JIRA?

For some context as FYI, for JDK 11 support, I asked feasibility about Hive 2.3.6 release at this Hive dev thread. Thankfully, seems positive. @wangyum quickly started worked on that and @alanfgates (from Hive) is actively cooperative - thanks again.

Therefore, two communities are cooperating at SPARK-28684 and HIVE-22096 . It might be much easier if we can test JDK 11 in PR builder in Spark side.

[shaneknapp]

@shaneknapp .. actually, adding test-java11 in PR builder turned to be a bit important now.. @wangyum, mind filing an JIRA?

it'll be easy to add the java11 check in run-tests-jenkins.py but possibly a bit more difficult to munge paths to ensure we're on the right java.

i'll start hacking on it now while i wait for the jira to pop up.

[shaneknapp]

i'll start hacking on it now while i wait for the jira to pop up.

i created the jira: https://issues.apache.org/jira/browse/SPARK-28701

pull request coming now.