Skip to content

HBASE-22749: Distributed MOB compactions #921

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
VladRodionov wants to merge 17 commits into from
Closed

HBASE-22749: Distributed MOB compactions #921

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
9d20cca
HBASE-22749: Distributed MOB compactions
Dec 9, 2019
a09e6ff
HBASE-22749: Distributed MOB compactions. (fixed UTs and code clean up)
Dec 13, 2019
0855b41
HBASE-22749: Distributed MOB compactions (some comments addressed)
Jan 13, 2020
d56413d
HBASE-22749: code review
Jan 24, 2020
1678d72
HBASE-22749 fix missing import for TestBulkLoadReplication
busbey Jan 31, 2020
9365f40
fixed compilation issue
VladRodionov Feb 4, 2020
50df62b
fixed failing mob tests
VladRodionov Feb 5, 2020
ea939c5
fixed checkstyle, findbugs, whitespaces
VladRodionov Feb 5, 2020
2ad64c0
small findbug fix
VladRodionov Feb 5, 2020
72dafd1
small findbug fix again
VladRodionov Feb 6, 2020
29fcdc6
small findbug fix again
VladRodionov Feb 6, 2020
064aa05
addressed review comments
VladRodionov Feb 12, 2020
54cce91
fixed checkstyle and findbugs
VladRodionov Feb 13, 2020
42bd683
HBASE-23840: revert IO optiized mode back to general during migration
VladRodionov Feb 14, 2020
8ee22c2
Abort MOB file cleaner chore in case of CF failure
VladRodionov Feb 14, 2020
e11ec68
whitespace checkstyle
VladRodionov Feb 15, 2020
e02bc40
whitespace checkstyle
VladRodionov Feb 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions hbase-checkstyle/src/main/resources/hbase/checkstyle-suppressions.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,4 +55,5 @@
<suppress checks="HideUtilityClassConstructor" files="org.apache.hadoop.hbase.util.ByteRangeUtils"/>
<!-- Will not be final, because it is InterfaceAudience.Public -->
<suppress checks="FinalClass" files="org.apache.hadoop.hbase.net.Address"/>
<suppress checks="MethodLength" files="org.apache.hadoop.hbase.mob.FaultyMobStoreCompactor.java"/>
</suppressions>
49 changes: 8 additions & 41 deletions hbase-common/src/main/resources/hbase-default.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1814,43 +1814,24 @@ possible configurations would overwhelm and obscure the important.
</description>
</property>
<property>
<name>hbase.master.mob.ttl.cleaner.period</name>
<name>hbase.master.mob.cleaner.period</name>
<value>86400</value>
<description>
The period that ExpiredMobFileCleanerChore runs. The unit is second.
The period that MobFileCleanerChore runs. The unit is second.
The default value is one day. The MOB file name uses only the date part of
the file creation time in it. We use this time for deciding TTL expiry of
the files. So the removal of TTL expired files might be delayed. The max
delay might be 24 hrs.
</description>
</property>
<property>
<name>hbase.mob.compaction.mergeable.threshold</name>
<value>1342177280</value>
<description>
If the size of a mob file is less than this value, it's regarded as a small
file and needs to be merged in mob compaction. The default value is 1280MB.
</description>
</property>
<property>
<name>hbase.mob.delfile.max.count</name>
<value>3</value>
<description>
The max number of del files that is allowed in the mob compaction.
In the mob compaction, when the number of existing del files is larger than
this value, they are merged until number of del files is not larger this value.
The default value is 3.
</description>
</property>
<property>
<name>hbase.mob.compaction.batch.size</name>
<value>100</value>
<name>hbase.mob.major.compaction.region.batch.size</name>
<value>0</value>
<description>
The max number of the mob files that is allowed in a batch of the mob compaction.
The mob compaction merges the small mob files to bigger ones. If the number of the
small files is very large, it could lead to a "too many opened file handlers" in the merge.
And the merge has to be split into batches. This value limits the number of mob files
that are selected in a batch of the mob compaction. The default value is 100.
The max number of a MOB table regions that is allowed in a batch of the mob compaction. By
setting this number to a custom value, users can control the overall effect of a major
compaction of a large MOB-enabled table.
Default is 0 - means no limit - all regions of a MOB table will be compacted at once
</description>
</property>
<property>
Expand All @@ -1861,20 +1842,6 @@ possible configurations would overwhelm and obscure the important.
The default value is one week.
</description>
</property>
<property>
<name>hbase.mob.compactor.class</name>
<value>org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactor</value>
<description>
Implementation of mob compactor, the default one is PartitionedMobCompactor.
</description>
</property>
<property>
<name>hbase.mob.compaction.threads.max</name>
<value>1</value>
<description>
The max number of threads used in MobCompactor.
</description>
</property>
<property>
<name>hbase.snapshot.master.timeout.millis</name>
<value>300000</value>
Expand Down
Loading