Skip to content

HBASE-24601: Change default Hfile storage policy from HOT to NONE for HDFS #1940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Apache9 merged 1 commit into from
Oct 2, 2021
Merged

HBASE-24601: Change default Hfile storage policy from HOT to NONE for HDFS #1940

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,8 @@ public class HStore implements Store, HeapSize, StoreConfigInformation,
"hbase.server.compactchecker.interval.multiplier";
public static final String BLOCKING_STOREFILES_KEY = "hbase.hstore.blockingStoreFiles";
public static final String BLOCK_STORAGE_POLICY_KEY = "hbase.hstore.block.storage.policy";
// keep in accordance with HDFS default storage policy
public static final String DEFAULT_BLOCK_STORAGE_POLICY = "HOT";
// "NONE" is not a valid storage policy and means we defer the policy to HDFS
public static final String DEFAULT_BLOCK_STORAGE_POLICY = "NONE";
Copy link
Contributor

@anoopsjohn anoopsjohn Jun 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Referring to the patch for WAL dir policy fix, is this enough? In case of NONE we should avoid calling the HDFS API or else will see unwanted exception logs.

Copy link
Member Author

@imbajin imbajin Jun 21, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated JIRA for more details

  1. Using the HOT or NONE strategy currently won't throw an unexpected exception because HBASE-20691 has already covered the different situation. And they invoke the same method, that's why we can use NONE directly.
  2. It makes sense to allow Hbase users to set the storage strategy corresponding to the Hfile/WAL directory, because most users may not be familiar with how to operate the heterogeneous commands of HDFS
  3. However, if the default value stored in Hfile is still HOT, there will be potential semantic conflicts. HDFS managers and Hbase users may not know this default setting and will not modify it, but it actually changes HDFS Storage strategy.
    This is a omission point of HBASE-20691, It only modified the default policy of WAL, but forgot to modify Hfile.

TBH, it's' a legacy issue to decide if we should avoid/ban setStoragePolicy() in Hbase, seems that keeping the status quo has the least impact now (ban the whole reflect API for HDFS should be done in a separate patch)

public static final int DEFAULT_COMPACTCHECKER_INTERVAL_MULTIPLIER = 1000;
public static final int DEFAULT_BLOCKING_STOREFILE_COUNT = 16;

Expand Down