Add background job to clear unreferenced state groups#18154
Merged
Conversation
Member
Author
|
I still need to add tests, but otherwise this is currently working. |
devonh
commented
Feb 12, 2025
Co-authored-by: Erik Johnston <[email protected]>
Co-authored-by: Erik Johnston <[email protected]>
Co-authored-by: Erik Johnston <[email protected]>
Co-authored-by: Erik Johnston <[email protected]>
Member
erikjohnston
left a comment
erikjohnston
left a comment
There was a problem hiding this comment.
Just a couple of minor tweaks, otherwise I think the approach is good
Co-authored-by: Erik Johnston <[email protected]>
Member
Author
|
Should be good to go now! |
erikjohnston
approved these changes
Feb 25, 2025
Member
|
Woop, thanks! Sorry for it going round in circles a bit |
Co-authored-by: Erik Johnston <[email protected]>
reivilibre
added a commit
that referenced
this pull request
Mar 7, 2025
reivilibre
added a commit
that referenced
this pull request
Mar 7, 2025
…s introduced in v1.126.0rc1), due to a suspected issue that causes increased disk usage. (#18222) Revert "Add background job to clear unreferenced state groups (#18154)" This mechanism is suspected of inserting large numbers of rows into `state_groups_state`, thus unreasonably increasing disk usage. See: #18217 This reverts commit 5121f92 (#18154). --------- Signed-off-by: Olivier 'reivilibre <[email protected]>
|
So, now that this has been reverted because of #18217, is there a plan to rewrite this feature? Smaller instances like ours desperately need this to continue operating. Maybe it is just a matter of doing it in batches of limited size? |
Member
Author
This feature will be fixed to avoid the issue with increasing state group state rows. |
Member
Author
|
New PR: #18254 |
3 tasks
devonh
added a commit
that referenced
this pull request
Mar 21, 2025
This PR fixes #18154 to avoid de-deltaing state groups which resulted in DB size temporarily increasing until the DB was `VACUUM`'ed. As a result, less state groups will get deleted now. It also attempts to improve performance by not duplicating work when processing state groups it has already processed in previous iterations. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #18150
Pull Request Checklist
EventStoretoEventWorkerStore.".code blocks.(run the linters)