feat: fix gc objects #1435
Merged
feat: fix gc objects #1435
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
clydemeng
reviewed
Nov 15, 2024
modular/executor/execute_gc.go
Outdated
| pieceKey = e.baseApp.PieceOp().SegmentPieceKey(objectInfo.Id.Uint64(), segIdx, objectVersion) | ||
| } | ||
| // ignore this delete api error, TODO: refine gc workflow by enrich metadata index. | ||
| deleteErr := e.baseApp.PieceStore().DeletePiece(ctx, pieceKey) |
There was a problem hiding this comment.
can we we "DeletePiecesByPrefix" in this case as well?
There was a problem hiding this comment.
Done. Updated both primary sp & secondary sp's delete logic to batch delete pieces with object prefix.
annielz
force-pushed
the
feat-fix-gc-objects
branch
2 times, most recently
from
2a6a3d1 to
70764e3
Compare
feat: lint feat: lint
annielz
force-pushed
the
feat-fix-gc-objects
branch
from
70764e3 to
507463c
Compare
feat: re-organize size calc for disk store feat: remove debug log fix: pointer
annielz
force-pushed
the
feat-fix-gc-objects
branch
from
7a3d8a0 to
02f6932
Compare
clydemeng
approved these changes
Nov 18, 2024
ruojunm
approved these changes
Nov 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently in sp, if an object is deleted within the same tx as its local virtual group (lvg), the async GC Task will have issue finding its secondary sps, as this info is only available by using logic objectId -> lvg -> gvg -> secondary sps.
In this case, it used to be that the data in secondary sp's data storage will not be removed. Now updating logic for all sps to check for files that have prefix related to the object (as "e{objectid}_"), and remove all related files.
Rationale
Update logic of GC
Example
https://testnet.greenfieldscan.com/tx/0x2a34610146b15f60728e79b96181dc2af60825578822f70b913348b6cb8bed28
This link is an example of object & lvg in the same tx.
Changes
Notable changes:
Potential Impacts