HBASE-26827 RegionServer JVM crash when compact mob table #4206

mymeiyi · 2022-03-11T08:45:15Z

No description provided.

GeorryHuang · 2022-03-11T09:15:43Z

Great! LGTM.

Apache-HBase · 2022-03-11T09:32:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	6m 29s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 47s	master passed
+1 💚	compile	2m 22s	master passed
+1 💚	checkstyle	0m 36s	master passed
+1 💚	spotbugs	1m 20s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 24s	the patch passed
+1 💚	compile	2m 15s	the patch passed
+1 💚	javac	2m 15s	the patch passed
+1 💚	checkstyle	0m 35s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 35s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	1m 23s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		37m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4206
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 2d3125b43144 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `71ff3ec`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	60 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-11T14:29:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 43s	master passed
+1 💚	compile	0m 35s	master passed
+1 💚	shadedjars	3m 50s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 22s	the patch passed
+1 💚	compile	0m 34s	the patch passed
+1 💚	javac	0m 34s	the patch passed
+1 💚	shadedjars	3m 50s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
+1 💚	unit	316m 30s	hbase-server in the patch passed.
		334m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4206
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux bba82759156f 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `71ff3ec`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/1/testReport/
Max. process+thread count	3760 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-11T17:18:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 19s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	7m 9s	master passed
+1 💚	compile	1m 54s	master passed
+1 💚	shadedjars	7m 45s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 6s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	6m 20s	the patch passed
+1 💚	compile	1m 39s	the patch passed
+1 💚	javac	1m 39s	the patch passed
+1 💚	shadedjars	7m 13s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 53s	the patch passed
		_ Other Tests _
+1 💚	unit	463m 19s	hbase-server in the patch passed.
		501m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4206
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ee69bdcd2245 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `71ff3ec`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/1/testReport/
Max. process+thread count	1832 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2022-03-12T15:13:53Z

So here we can only release all the cells at onceafter shipping them all? Will it cause OOM?

Apache9 · 2022-07-01T14:25:37Z

Any updates here? @mymeiyi @GeorryHuang

mymeiyi · 2022-07-07T02:47:48Z

So here we can only release all the cells at onceafter shipping them all? Will it cause OOM?

Sorry for so late to reply. This problem is also disscussed in #3318 (comment).

the block release may be delayed. There is a configuration named "hbase.hstore.compaction.kv.max" and default value is 10 to limit scan cell count in compaction. But if a cell size is huge, we can not avoid the delay.

Now this is the most simple way to fix it. We may need to find a better way if we want to release timely.

Apache-HBase · 2022-07-08T13:13:13Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 11s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 30s	master passed
+1 💚	compile	2m 18s	master passed
+1 💚	checkstyle	0m 32s	master passed
+1 💚	spotless	0m 44s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 20s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 11s	the patch passed
+1 💚	compile	2m 16s	the patch passed
+1 💚	javac	2m 16s	the patch passed
+1 💚	checkstyle	0m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 34s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 24s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		32m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4206
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 74c61516b1af 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2197b38`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-08T16:19:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 42s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 21s	master passed
+1 💚	compile	0m 33s	master passed
+1 💚	shadedjars	3m 53s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 7s	the patch passed
+1 💚	compile	0m 33s	the patch passed
+1 💚	javac	0m 33s	the patch passed
+1 💚	shadedjars	3m 56s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
+1 💚	unit	202m 56s	hbase-server in the patch passed.
		218m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4206
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 201902bf7b22 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2197b38`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/2/testReport/
Max. process+thread count	2158 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-08T16:49:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 7s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 22s	master passed
+1 💚	compile	0m 56s	master passed
+1 💚	shadedjars	4m 38s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 33s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 10s	the patch passed
+1 💚	compile	0m 53s	the patch passed
+1 💚	javac	0m 53s	the patch passed
+1 💚	shadedjars	4m 28s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 28s	the patch passed
		_ Other Tests _
+1 💚	unit	226m 27s	hbase-server in the patch passed.
		248m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4206
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ab938921ed74 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2197b38`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/2/testReport/
Max. process+thread count	2626 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4206/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

GeorryHuang approved these changes Mar 11, 2022

View reviewed changes

HBASE-26827 RegionServer JVM crash when compact mob table

15e6f2b

mymeiyi force-pushed the mob-Compactor branch from c43eafe to 15e6f2b Compare July 8, 2022 12:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-26827 RegionServer JVM crash when compact mob table #4206

HBASE-26827 RegionServer JVM crash when compact mob table #4206

mymeiyi commented Mar 11, 2022

Uh oh!

GeorryHuang commented Mar 11, 2022

Uh oh!

Apache-HBase commented Mar 11, 2022

Uh oh!

Apache-HBase commented Mar 11, 2022

Uh oh!

Apache-HBase commented Mar 11, 2022

Uh oh!

Apache9 commented Mar 12, 2022

Uh oh!

Apache9 commented Jul 1, 2022

Uh oh!

mymeiyi commented Jul 7, 2022 •

edited

Loading

Uh oh!

Apache-HBase commented Jul 8, 2022

Uh oh!

Apache-HBase commented Jul 8, 2022

Uh oh!

Apache-HBase commented Jul 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HBASE-26827 RegionServer JVM crash when compact mob table #4206

Are you sure you want to change the base?

HBASE-26827 RegionServer JVM crash when compact mob table #4206

Conversation

mymeiyi commented Mar 11, 2022

Uh oh!

GeorryHuang commented Mar 11, 2022

Uh oh!

Apache-HBase commented Mar 11, 2022

Uh oh!

Apache-HBase commented Mar 11, 2022

Uh oh!

Apache-HBase commented Mar 11, 2022

Uh oh!

Apache9 commented Mar 12, 2022

Uh oh!

Apache9 commented Jul 1, 2022

Uh oh!

mymeiyi commented Jul 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache-HBase commented Jul 8, 2022

Uh oh!

Apache-HBase commented Jul 8, 2022

Uh oh!

Apache-HBase commented Jul 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mymeiyi commented Jul 7, 2022 •

edited

Loading