HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor #3234
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
This change is moved here from #3230. We know from the test report there that |
Contributor
Author
|
@Apache9 added some context on #3230
|
Contributor
Author
|
Although the unit test will fail, when I tried this change out on a cluster in a very write heavy ingestion test, the end result was good. The ingestion test completes successfully and without weird artifacts in the logging. All split regions are GCed by procedures. In memory state aligns with filesystem state. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
saintstack
approved these changes
May 5, 2021
Contributor
saintstack
left a comment
saintstack
left a comment
There was a problem hiding this comment.
Yes to one-place only (in the procedure).
I've lost context on why the CJ redundant removes. I defer to the test you ran Andrew at scale.
(I just noticed the wonky split message -- have started a bit of ITBLL over here...)
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
I've committed a few changes over the past couple of days that address two issues here. One was an accumulation of SPLIT state regionStates for as long as CatalogJanitor is deferring cleanup, because daughters still have references, because compaction is backed up. The accumulation is fine but it caused concerning warnings at each balancer iteration. Another was a race condition between split handling and regionserver report handling that could cause multiple split procedures to get scheduled for the same split request. Only one could succeed. The others would cause log noise but their failures were harmless. Would be good to get a second opinion. |
Contributor
Author
|
Ok, let me rebase this and fix the unit tests. |
…itor In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both of these procedures clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures.
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
Actual failing test |
|
🎊 +1 overall
This message was automatically generated. |
Apache9
approved these changes
May 6, 2021
virajjasani
approved these changes
May 6, 2021
Contributor
virajjasani
left a comment
virajjasani
left a comment
There was a problem hiding this comment.
So we are just removing what scheduled Procs are anyways going to do right, sounds good. +1
asfgit
pushed a commit
that referenced
this pull request
May 6, 2021
…itor (#3234) In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both of these procedures clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures. Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Michael Stack <[email protected]> Signed-off-by: Viraj Jasani <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
May 6, 2021
…itor (#3234) In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both of these procedures clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures. Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Michael Stack <[email protected]> Signed-off-by: Viraj Jasani <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both of these procedures clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. This is least a nit, but probably a source of future bugs. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures.