HBASE-22749: Distributed MOB compactions #921

VladRodionov · 2019-12-09T20:02:08Z

Finalizing HBASE-22749. Rebased patch against current HBase master, created new pull PR

busbey · 2019-12-09T20:33:52Z

just to be clear, this includes all the follow-on commits from #623 as well right? i.e. HBASE-23209, HBASE-23188, HBASE-23266, HBASE-23267, and HBASE-23189?

VladRodionov · 2019-12-09T20:48:21Z

Yes, it includes all from PR #623 + some fixes to make this patch mergeable with current HBase master.

busbey · 2019-12-10T00:26:13Z

Please clean up qabot flagged issues. The unit test failures look related at first glance.

busbey

review in progress.

hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestMobCompaction.java

hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCleanerChore.java

busbey

still WIP

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MobFileCompactionChore.java

busbey

more WIP

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobConstants.java

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java

hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java

busbey

finished getting through all the changes. yay!

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TesMobFileCleanerChore.java

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompaction.java

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionBase.java

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobFileName.java

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMobStoreCompaction.java

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCompactionChore.java

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TesMobFileCleanerChore.java

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionOptRegionBatchMode.java

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionRegularMode.java

...erver/src/test/java/org/apache/hadoop/hbase/mob/TestMobCompactionRegularRegionBatchMode.java

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestMobFileCleanerChore.java

busbey · 2020-02-15T00:45:56Z

a precommit check of e11ec68 is in progress; about 20 minutes into the usual 2.5 hours.

busbey

QA bot should be back in a couple of hours, I'd expect with a clean run.

I'm going to push a branch with these changes so we can get a full nightly run with the changes in place. Once I get that clean I'll move forward with merging to master.

Thanks for sticking through on this large improvement @VladRodionov!

Apache-HBase · 2020-02-15T23:18:42Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 43s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 23 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	0m 41s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 5s	master passed
+1 💚	compile	2m 25s	master passed
+1 💚	checkstyle	3m 0s	master passed
+0 🆗	refguide	6m 23s	branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	shadedjars	6m 1s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 37s	master passed
+0 🆗	spotbugs	5m 7s	Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	0m 17s	branch/hbase-checkstyle no findbugs output file (findbugsXml.xml)
-0 ⚠️	patch	5m 38s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for patch
+1 💚	mvninstall	6m 33s	the patch passed
+1 💚	compile	2m 24s	the patch passed
+1 💚	javac	2m 24s	the patch passed
+1 💚	checkstyle	2m 59s	root: The patch generated 0 new + 427 unchanged - 58 fixed = 427 total (was 485)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 4s	The patch has no ill-formed XML file.
+0 🆗	refguide	6m 25s	patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	shadedjars	6m 1s	patch has no errors when building our shaded downstream artifacts.
+1 💚	hadoopcheck	20m 10s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚	javadoc	1m 26s	the patch passed
+0 🆗	findbugs	0m 14s	hbase-checkstyle has no data from findbugs
		_ Other Tests _
+1 💚	unit	0m 14s	hbase-checkstyle in the patch passed.
+1 💚	unit	1m 3s	hbase-common in the patch passed.
+1 💚	unit	66m 16s	hbase-server in the patch passed.
+1 💚	unit	1m 16s	hbase-it in the patch passed.
+1 💚	asflicense	2m 15s	The patch does not generate ASF License warnings.
		165m 14s

Subsystem	Report/Notes
Docker	Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/artifact/out/Dockerfile
GITHUB PR	#921
JIRA Issue	HBASE-22749
Optional Tests	dupname asflicense checkstyle javac javadoc unit xml refguide spotbugs findbugs shadedjars hadoopcheck hbaseanti compile
uname	Linux 514b8f7656b0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-921/out/precommit/personality/provided.sh
git revision	master / `ae781e5`
Default Java	1.8.0_181
refguide	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/artifact/out/branch-site/book.html
refguide	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/artifact/out/patch-site/book.html
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/testReport/
Max. process+thread count	8774 (vs. ulimit of 10000)
modules	C: hbase-checkstyle hbase-common hbase-server hbase-it U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-921/14/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

busbey · 2020-02-18T18:50:55Z

qabot looks great. I got some noise on my nightly branch. checking on it today.

busbey · 2020-02-19T20:54:44Z

nightly looks good as well. I'll post details about that to jira and then move forward on merging.

thanks for chasing this down @VladRodionov

busbey · 2020-02-19T23:23:29Z

merged via squashed commit b8194b4

busbey self-requested a review December 9, 2019 20:39

This comment has been minimized.

Sign in to view

busbey requested changes Dec 12, 2019

View reviewed changes

This comment has been minimized.

Sign in to view

busbey requested changes Dec 16, 2019

View reviewed changes

busbey requested changes Dec 17, 2019

View reviewed changes

busbey requested changes Dec 19, 2019

View reviewed changes

busbey requested changes Jan 9, 2020

View reviewed changes

busbey requested changes Jan 14, 2020

View reviewed changes

busbey force-pushed the master branch from 33e48db to 3e60b61 Compare January 14, 2020 18:13

This comment has been minimized.

Sign in to view

VladRodionov force-pushed the master branch 2 times, most recently from 926fcff to f7dc66d Compare January 24, 2020 18:46

busbey closed this Feb 2, 2020

busbey force-pushed the master branch from f7dc66d to ba9f026 Compare February 2, 2020 18:30

busbey reopened this Feb 3, 2020

Vladimir Rodionov and others added 5 commits February 3, 2020 17:21

HBASE-22749: Distributed MOB compactions

9d20cca

HBASE-22749: Distributed MOB compactions. (fixed UTs and code clean up)

a09e6ff

HBASE-22749: Distributed MOB compactions (some comments addressed)

0855b41

HBASE-22749: code review

d56413d

HBASE-22749 fix missing import for TestBulkLoadReplication

1678d72

VladRodionov force-pushed the master branch from 94e0a90 to 1678d72 Compare February 4, 2020 01:29

This comment has been minimized.

Sign in to view

fixed compilation issue

9365f40

fixed failing mob tests

50df62b

This comment has been minimized.

Sign in to view

VladRodionov added 2 commits February 5, 2020 13:40

fixed checkstyle, findbugs, whitespaces

ea939c5

small findbug fix

2ad64c0

This comment has been minimized.

Sign in to view

small findbug fix again

72dafd1

This comment has been minimized.

Sign in to view

small findbug fix again

29fcdc6

This comment has been minimized.

Sign in to view

busbey requested changes Feb 6, 2020

View reviewed changes

addressed review comments

064aa05

This comment has been minimized.

Sign in to view

fixed checkstyle and findbugs

54cce91

This comment has been minimized.

Sign in to view

VladRodionov added 2 commits February 14, 2020 11:18

HBASE-23840: revert IO optiized mode back to general during migration

42bd683

Abort MOB file cleaner chore in case of CF failure

8ee22c2

This comment has been minimized.

Sign in to view

whitespace checkstyle

e11ec68

This comment has been minimized.

Sign in to view

busbey approved these changes Feb 15, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

whitespace checkstyle

e02bc40

This comment has been minimized.

Sign in to view

busbey closed this Feb 19, 2020

HBASE-22749: Distributed MOB compactions #921

HBASE-22749: Distributed MOB compactions #921

Uh oh!

Conversation

VladRodionov commented Dec 9, 2019

Uh oh!

busbey commented Dec 9, 2019

Uh oh!

VladRodionov commented Dec 9, 2019

Uh oh!

This comment has been minimized.

busbey commented Dec 10, 2019

Uh oh!

busbey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

busbey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

busbey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

busbey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!