HBASE-24007 Get `-PrunLargeTests` passing on JDK11 #1351

ndimiduk · 2020-03-25T21:10:25Z

Minor tweaks required to get passing runs of -PrunLargeTests.

Minimum Hadoop version is 3.2.0 due to HADOOP-12760.
JDK11 looks like it consumes more memory than JDK8, so failures due to OOME see more common here. Bumping heap allocated to surefire forks allows better pass rate.

Minor tweaks required to get passing runs of `-PrunLargeTests`. * Minimum Hadoop version is 3.2.0 due to [HADOOP-12760](https://issues.apache.org/jira/browse/HADOOP-12760). * JDK11 looks like it consumes more memory than JDK8, so failures due to OOME see more common here. Bumping heap allocated to surefire forks allows better pass rate.

ndimiduk · 2020-03-25T21:11:40Z

Anyone know of a better way of managing the minimum hadoop version dependency? Some enforcer-plugin foo?

jojochuang · 2020-03-25T21:23:11Z

We've seen performance issue with JDK11 too. They took extra time to run that breaks our internal functional tests. Later we realized it's because JDK11 defaults to G1GC and thus the memory consumption is different/more

Apache-HBase · 2020-03-25T21:36:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 17s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
		_ Patch Compile Tests _
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	12m 39s	Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2.
		_ Other Tests _
+1 💚	asflicense	0m 16s	The patch does not generate ASF License warnings.
		21m 33s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1351
Optional Tests	dupname asflicense hadoopcheck xml
uname	Linux be55078c58c4 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5e63509`
Max. process+thread count	63 (vs. ulimit of 10000)
modules	C: . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-03-25T22:03:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	8m 58s	master passed
+1 💚	compile	5m 8s	master passed
+1 💚	shadedjars	7m 57s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 24s	root in master failed.
		_ Patch Compile Tests _
+1 💚	mvninstall	8m 27s	the patch passed
+1 💚	compile	5m 17s	the patch passed
+1 💚	javac	5m 17s	the patch passed
+1 💚	shadedjars	7m 49s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 21s	root in the patch failed.
		_ Other Tests _
-0 ⚠️	unit	2m 8s	root in the patch failed.
		48m 27s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1351
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 458afc272f14 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5e63509`
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/testReport/
Max. process+thread count	487 (vs. ulimit of 10000)
modules	C: . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-03-25T22:05:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 9s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	6m 29s	master passed
+1 💚	compile	3m 33s	master passed
+1 💚	shadedjars	5m 33s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 47s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 53s	the patch passed
+1 💚	compile	3m 29s	the patch passed
+1 💚	javac	3m 29s	the patch passed
+1 💚	shadedjars	5m 32s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 47s	the patch passed
		_ Other Tests _
-1 ❌	unit	11m 55s	root in the patch failed.
		50m 32s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#1351
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 225d91121d52 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5e63509`
Default Java	1.8.0_232
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-root.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/testReport/
Max. process+thread count	870 (vs. ulimit of 10000)
modules	C: . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1351/1/console
versions	git=2.17.1 maven=2018-06-17T18:33:14Z)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2020-03-25T23:05:46Z

TestVerifyBucketCacheFile strikes again :'(

ndimiduk · 2020-03-27T19:27:38Z

I was asked offline: the dependency on HADOOP-12760 comes from TestSaslFanOutOneBlockAsyncDFSOutput, which has been on our flakey exclude list for a while.

ndimiduk · 2020-03-27T19:28:13Z

Bump review request please.

HorizonNet

LGTM. Let's give it a try.

ndimiduk · 2020-03-30T16:31:28Z

Thanks @HorizonNet !

Minor tweaks required to get passing runs of `-PrunLargeTests`. * Minimum Hadoop version is 3.2.0 due to [HADOOP-12760](https://issues.apache.org/jira/browse/HADOOP-12760). * JDK11 looks like it consumes more memory than JDK8, so failures due to OOME see more common here. Bumping heap allocated to surefire forks allows better pass rate. Signed-off-by: Jan Hentschel <[email protected]>

ndimiduk requested review from HorizonNet, busbey and saintstack March 25, 2020 21:11

ndimiduk requested review from Apache9 and infraio March 27, 2020 19:28

HorizonNet approved these changes Mar 28, 2020

View reviewed changes

ndimiduk merged commit d77dbef into apache:master Mar 30, 2020

ndimiduk deleted the 24007-largeTests-jdk11 branch March 30, 2020 16:31

wchevreuil mentioned this pull request Aug 26, 2025

HBASE-28463 Time Based Priority for BucketCache #7192

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-24007 Get `-PrunLargeTests` passing on JDK11 #1351

HBASE-24007 Get `-PrunLargeTests` passing on JDK11 #1351

Uh oh!

ndimiduk commented Mar 25, 2020

Uh oh!

ndimiduk commented Mar 25, 2020

Uh oh!

jojochuang commented Mar 25, 2020

Uh oh!

Apache-HBase commented Mar 25, 2020

Uh oh!

Apache-HBase commented Mar 25, 2020

Uh oh!

Apache-HBase commented Mar 25, 2020

Uh oh!

ndimiduk commented Mar 25, 2020

Uh oh!

ndimiduk commented Mar 27, 2020

Uh oh!

ndimiduk commented Mar 27, 2020

Uh oh!

HorizonNet left a comment

Uh oh!

ndimiduk commented Mar 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HBASE-24007 Get -PrunLargeTests passing on JDK11 #1351

HBASE-24007 Get -PrunLargeTests passing on JDK11 #1351

Uh oh!

Conversation

ndimiduk commented Mar 25, 2020

Uh oh!

ndimiduk commented Mar 25, 2020

Uh oh!

jojochuang commented Mar 25, 2020

Uh oh!

Apache-HBase commented Mar 25, 2020

Uh oh!

Apache-HBase commented Mar 25, 2020

Uh oh!

Apache-HBase commented Mar 25, 2020

Uh oh!

ndimiduk commented Mar 25, 2020

Uh oh!

ndimiduk commented Mar 27, 2020

Uh oh!

ndimiduk commented Mar 27, 2020

Uh oh!

HorizonNet left a comment

Choose a reason for hiding this comment

Uh oh!

ndimiduk commented Mar 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HBASE-24007 Get `-PrunLargeTests` passing on JDK11 #1351

HBASE-24007 Get `-PrunLargeTests` passing on JDK11 #1351