HBASE-28804: Implement asynchronous retrieval of bucket-cache data from persistence #6182
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wchevreuil
reviewed
Aug 28, 2024
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java
Outdated
Show resolved
Hide resolved
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePreadReader.java
Outdated
Show resolved
Hide resolved
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
jhungund
force-pushed
the
HBASE-28804
branch
2 times, most recently
from
7853e54 to
8377eb9
Compare
wchevreuil
reviewed
Aug 29, 2024
| assertTrue(reader.prefetchStarted() || reader.prefetchComplete()); | ||
|
|
||
| // Added some delay as we have started the timer a bit late. | ||
| Thread.sleep(500); |
Contributor
There was a problem hiding this comment.
Why do we need this extra sleep after the loop above?
Contributor
Author
There was a problem hiding this comment.
This was present earlier along with the comment at line 378.
Contributor
Author
There was a problem hiding this comment.
I will try to remove this one and check if the test passes.
Contributor
There was a problem hiding this comment.
Yeah, but I assume now that we have an extra wait of 10000, we don't need this one anymore?
Comment on lines
370
to
375
| while (!reader.prefetchStarted() && !reader.prefetchComplete()) { | ||
| // Wait until the prefetch is triggered. | ||
| Thread.sleep(500); | ||
| assertTrue("Prefetch should start post configured delay", | ||
| getElapsedTime(startTime) > PrefetchExecutor.getPrefetchDelay()); | ||
| if (timeout <= 0) break; | ||
| timeout -= 500; | ||
| } |
Contributor
There was a problem hiding this comment.
Use Waiter.waitFor(Configuration conf, long timeout, long interval, Predicate<E> predicate)
Comment on lines
229
to
239
| long timeout = 120000; | ||
| try { | ||
| while (!cache.backingMap.containsKey(cacheKey) || cache.ramCache.containsKey(cacheKey)) { | ||
| Thread.sleep(100); | ||
| if (timeout <= 0) { | ||
| break; | ||
| } | ||
| timeout -= 100; | ||
| } | ||
| } finally { | ||
| Thread.sleep(1000); |
Contributor
There was a problem hiding this comment.
Use Waiter.waitFor(Configuration conf, long timeout, long interval, Predicate<E> predicate)
Contributor
Author
There was a problem hiding this comment.
Hi @wchevreuil , the config object is not available with us here. Probably I can leave out the change this. I had tried to change it to avoid infinite wait.
Contributor
There was a problem hiding this comment.
Just create one HBaseConfiguration.create() and pass along? Waiter only uses it to check for the hbase.test.wait.for.ratio property anyways.
Comment on lines
151
to
158
| long timeout = 120000; | ||
| while (!cache.backingMap.containsKey(cacheKey) || cache.ramCache.containsKey(cacheKey)) { | ||
| Thread.sleep(100); | ||
| if (timeout <= 0) { | ||
| break; | ||
| } | ||
| timeout -= 100; | ||
| } |
Contributor
There was a problem hiding this comment.
Use Waiter.waitFor(Configuration conf, long timeout, long interval, Predicate<E> predicate)
Contributor
Author
There was a problem hiding this comment.
Lot more changes needed to get config to this location. I will leave out this change for now.
Contributor
There was a problem hiding this comment.
We should add UTs for checking the behaviour of all methods changed as part of this async initialisation.
Contributor
Author
There was a problem hiding this comment.
I could validate all public methods. I am not sure if we can do it for private methods.
Contributor
There was a problem hiding this comment.
Public methods are enough.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
wchevreuil
reviewed
Sep 3, 2024
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
Outdated
Show resolved
Hide resolved
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
Outdated
Show resolved
Hide resolved
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java
Outdated
Show resolved
Hide resolved
jhungund
force-pushed
the
HBASE-28804
branch
from
9ba3087 to
d79f091
Compare
|
💔 -1 overall
This message was automatically generated. |
jhungund
force-pushed
the
HBASE-28804
branch
from
d79f091 to
52ee5e5
Compare
|
🎊 +1 overall
This message was automatically generated. |
jhungund
force-pushed
the
HBASE-28804
branch
from
52ee5e5 to
69295b8
Compare
|
💔 -1 overall
This message was automatically generated. |
jhungund
force-pushed
the
HBASE-28804
branch
from
69295b8 to
64b2a5e
Compare
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
retest |
…om persistence During the retrieval of data from bucket cache persistence file, a transient structure that stores the blocks ordered by filename is constructed from the backing map entries. The population of this transient structure is done during the server start-up. This process increases the region-server startup time, if the bucketcache has large number of blocks. This population happens inline with the server restart and blocks the server for several minutes. This makes the server restart inconvenient for the external users. Restarts during upgrade can run into timeout issues due to this delay in the server startup. Hence, the recommendation in this Jira is to make the cache-retrieval asynchronous to the server startup. During a server startup, a new thread is spawn that reads the persistence file and creates the required structures from persistence file. The server continues with the restart and does not wait for the bucket-cache initialisation to complete. Note that the bucket cache is not available immediately for usage and will only be ready to use after the data is repopulated from persistence into memory. The prefetch thread that may start before the bucket-cache is initialized is modified to wait until the bucket cache is initialized. Change-Id: I2c136d7b1d884f74642d29923172a1ad4ada36e4
Change-Id: I239c357a135a058650a20c5707c7c7303a248c85
Change-Id: Ie160b249f6a2bff18fd8a577ae32a263b7de25ea
…on state. Change-Id: Ib54f565152c391da727be7413dd775fb507daea7
Change-Id: I446475dc84d52403f67762b878af874b9cbe1937
Change-Id: I49d8e39cafff6cbe103f300235bec4aacd42934d
Change-Id: I8db2162c8237c05fd98d3c771f50e5a92c9b78b5
Change-Id: I391e5e453e077b7e5733cd997130467b356a3e66
Change-Id: Ieb6b5841cb7b23d919fbebbe420c0083167d6d0b
jhungund
force-pushed
the
HBASE-28804
branch
from
64b2a5e to
2c8438d
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
Hi @wchevreuil, I checked that the failing test is: `(base) janardhan.hungund@MacBook-Pro-4 hbase % git log
Hence, the failure does not seem to be related to this change. Thanks, |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
Hi @wchevreuil, The failing unit test, org.apache.hadoop.hbase.master.janitor.TestCatalogJanitor.testAlreadyRunningStatus does not seem to be related to this change. The local execution of the test passed on the same branch.
|
wchevreuil
approved these changes
Sep 9, 2024
jhungund
added a commit
to janardhanrh/hbase
that referenced
this pull request
Sep 9, 2024
…om persistence (apache#6182) Signed-off-by: Wellington Chevreuil <[email protected]> Change-Id: I515d81fdc484cd49308d3164067a2287b8901520
jhungund
added a commit
to janardhanrh/hbase
that referenced
this pull request
Sep 9, 2024
…om persistence (apache#6182) Signed-off-by: Wellington Chevreuil <[email protected]> Change-Id: I515d81fdc484cd49308d3164067a2287b8901520
wchevreuil
pushed a commit
that referenced
this pull request
Sep 9, 2024
…om persistence (#6182) Signed-off-by: Wellington Chevreuil <[email protected]>
wchevreuil
pushed a commit
that referenced
this pull request
Sep 11, 2024
…om persistence (#6182) (#6219) Signed-off-by: Wellington Chevreuil <[email protected]>
szucsvillo
pushed a commit
to szucsvillo/hbase
that referenced
this pull request
Feb 7, 2025
… bucket-cache data from persistence (apache#6182) (apache#6219) Signed-off-by: Wellington Chevreuil <[email protected]>
stoty
pushed a commit
to stoty/hbase
that referenced
this pull request
Nov 22, 2025
… bucket-cache data from persistence (apache#6182) (apache#6219) Signed-off-by: Wellington Chevreuil <[email protected]>
wchevreuil
pushed a commit
to wchevreuil/hbase
that referenced
this pull request
Dec 15, 2025
…om persistence (apache#6182) (apache#6219) The cherry-pick of HBASE-29727 into branch-2.6 accidently brought most of HBASE-28804 into branch-2, this is to backport the remaining original change in HBASE-28804. Signed-off-by: Wellington Chevreuil <[email protected]> Change-Id: I816141ccebf91191aaa4cb6afa04343b64663eb0
wchevreuil
pushed a commit
to wchevreuil/hbase
that referenced
this pull request
Dec 17, 2025
…om persistence (apache#6182) (apache#6219) The cherry-pick of HBASE-29727 into branch-2.6 accidently brought most of HBASE-28804 into branch-2, this is to backport the remaining original change in HBASE-28804. Signed-off-by: Wellington Chevreuil <[email protected]> Change-Id: I816141ccebf91191aaa4cb6afa04343b64663eb0
wchevreuil
pushed a commit
to wchevreuil/hbase
that referenced
this pull request
Dec 17, 2025
…om persistence (apache#6182) (apache#6219) The cherry-pick of HBASE-29727 into branch-2.6 accidently brought most of HBASE-28804 into branch-2, this is to backport the remaining original change in HBASE-28804. Signed-off-by: Wellington Chevreuil <[email protected]> Change-Id: I816141ccebf91191aaa4cb6afa04343b64663eb0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During the retrieval of data from bucket cache persistence file, a transient structure that stores the blocks ordered by filename is constructed from the backing map entries. The population of this transient structure is done during the server start-up. This process increases the region-server startup time, if the bucketcache has large number of blocks.
This population happens inline with the server restart and blocks the server for several minutes. This makes the server restart inconvenient for the external users. Restarts during upgrade can run into timeout issues due to this delay in the server startup.
Hence, the recommendation in this Jira is to make the cache-retrieval asynchronous to the server startup. During a server startup, a new thread is spawn that reads the persistence file and creates the required structures from persistence file. The server continues with the restart and does not wait for the bucket-cache initialisation to complete.
Note that the bucket cache is not available immediately for usage and will only be ready to use after the data is repopulated from persistence into memory.
The prefetch thread that may start before the bucket-cache is initialized is modified to wait until the bucket cache is initialized.
Change-Id: I2c136d7b1d884f74642d29923172a1ad4ada36e4