Remove unnecessary pictrs purge calls #5565

Nothing4You · 2025-03-29T04:17:47Z

This commit removes various cases in which Lemmy would purge images from pictrs, which would in many cases not be what admins intended.

pict-rs provides aliases on upload, which allows deduplicating multiple uploads of the same file in the backend, while still providing unique URLs for uploads.

As Lemmy used pictrs' purge API, this meant that not only the image alias referring to removed content was deleted, all other aliases were invalidated as well.

Additionally, even alias deletion would not appropriate in many of these cases, as they were lacking validation that they were exclusively used by the content they were supposed to get removed with.

This implements the following changes:

Purging a community no longer purges the community banner or icon from pict-rs.
Purging a community no longer purges all images referenced in post URLs and post thumbnails within the community from pict-rs.
Banning a user with content removal no longer purges their profile avatar or banner from pict-rs.
Banning a user with content removal no longer purges images referenced in post URLs and post thumbnails for all posts they created from pict-rs.
Banning a user with content removal no longer purges the community banners or icons for all communities they're the top mod of from pict-rs.
Banning a user with content removal now deletes all media they uploaded.
Purging a user no longer purges their profile avatar, banner, or images referenced in post URLs and post thumbnails for all posts they created from pict-rs. All media linked to their user account will still get deleted, which was already explicitly the case in the past.
On an NSFW-disabled instance, receiving a post update via federation, which changes the post from not NSFW to NSFW no longer purges post URL and thumbnail from pict-rs.

Some of the mentioned actions will still remove references to image URLs from the database, such as purging a community will still set its icon and banner to NULL in the db, but the associated images will no longer be purged from pict-rs.

As this stops erasure of thumbnails, #5564 has been created to ensure tracking the person that triggered the creation of thumbnails, which will allow removing them like other images.

The only remaining option to purge images attached to a post is now purging an individual post, which still erases the post URL and thumbnail from pict-rs entirely, including any other aliases. Purging and banning users with content removal will remove all aliases associated with them, which will end up deleting those images entirely when there are no other alias remaining.

fixes #5560

This commit removes various cases in which Lemmy would purge images from pictrs, which would in many cases not be what admins intended. pict-rs provides aliases on upload, which allows deduplicating multiple uploads of the same file in the backend, while still providing unique URLs for uploads. As Lemmy used pictrs' purge API, this meant that not only the image alias referring to removed content was deleted, all other aliases were invalidated as well. Additionally, even alias deletion would not appropriate in many of these cases, as they were lacking validation that they were exclusively used by the content they were supposed to get removed with. This implements the following changes: 1. Purging a community no longer purges the community banner or icon from pict-rs. 2. Purging a community no longer purges all images referenced in post URLs and post thumbnails within the community from pict-rs. 3. Banning a user with content removal no longer purges their profile avatar or banner from pict-rs. 4. Banning a user with content removal no longer purges images referenced in post URLs and post thumbnails for all posts they created from pict-rs. 5. Banning a user with content removal no longer purges the community banners or icons for all communities they're the top mod of from pict-rs. 6. Purging a user no longer purges their profile avatar, banner, or images referenced in post URLs and post thumbnails for all posts they created from pict-rs. All media linked to their user account will still get deleted, which was already explicitly the case in the past. Some of the mentioned actions will still remove references to image URLs from the database, such as banning a user with content removal setting their avatar and profile banner to `NULL` in the db, which prevents them from being displayed on their profile. As this stops erasure of thumbnails, LemmyNet#5564 has been created to ensure tracking the person that triggered the creation of thumbnails, which will allow removing them like other images. The only remaining option to purge images attached to a post is now purging an individual post, which still erases the post URL and thumbnail from pict-rs entirely, including any other aliases. Purging users will remove all aliases associated with them, which will end up deleting those images entirely when there are no other alias remaining. backports LemmyNet#5565 to 0.19 LemmyNet#5565 LemmyNet@988638d

Also add the changes to from LemmyNet#5565 related to handling NSFW posts from federation.

This commit removes various cases in which Lemmy would purge images from pictrs, which would in many cases not be what admins intended. pict-rs provides aliases on upload, which allows deduplicating multiple uploads of the same file in the backend, while still providing unique URLs for uploads. As Lemmy used pictrs' purge API, this meant that not only the image alias referring to removed content was deleted, all other aliases were invalidated as well. Additionally, even alias deletion would not appropriate in many of these cases, as they were lacking validation that they were exclusively used by the content they were supposed to get removed with. This implements the following changes: 1. Purging a community no longer purges the community banner or icon from pict-rs. 2. Purging a community no longer purges all images referenced in post URLs and post thumbnails within the community from pict-rs. 3. Banning a user with content removal no longer purges their profile avatar or banner from pict-rs. 4. Banning a user with content removal no longer purges images referenced in post URLs and post thumbnails for all posts they created from pict-rs. 5. Banning a user with content removal no longer purges the community banners or icons for all communities they're the top mod of from pict-rs. 6. Purging a user no longer purges their profile avatar, banner, or images referenced in post URLs and post thumbnails for all posts they created from pict-rs. All media linked to their user account will still get deleted, which was already explicitly the case in the past. Some of the mentioned actions will still remove references to image URLs from the database, such as banning a user with content removal setting their avatar and profile banner to `NULL` in the db, which prevents them from being displayed on their profile. As this stops erasure of thumbnails, LemmyNet#5564 has been created to ensure tracking the person that triggered the creation of thumbnails, which will allow removing them like other images. The only remaining option to purge images attached to a post is now purging an individual post, which still erases the post URL and thumbnail from pict-rs entirely, including any other aliases. Purging users will remove all aliases associated with them, which will end up deleting those images entirely when there are no other alias remaining. backports LemmyNet#5565 to 0.19 LemmyNet#5565 LemmyNet@988638d

crates/api/src/site/purge/community.rs

crates/db_schema/src/impls/post.rs

crates/api_common/src/utils.rs

This commit removes various cases in which Lemmy would purge images from pictrs, which would in many cases not be what admins intended. pict-rs provides aliases on upload, which allows deduplicating multiple uploads of the same file in the backend, while still providing unique URLs for uploads. As Lemmy used pictrs' purge API, this meant that not only the image alias referring to removed content was deleted, all other aliases were invalidated as well. Additionally, even alias deletion would not appropriate in many of these cases, as they were lacking validation that they were exclusively used by the content they were supposed to get removed with. This implements the following changes: 1. Purging a community no longer purges the community banner or icon from pict-rs. 2. Purging a community no longer purges all images referenced in post URLs and post thumbnails within the community from pict-rs. 3. Banning a user with content removal no longer purges their profile avatar or banner from pict-rs. 4. Banning a user with content removal no longer purges images referenced in post URLs and post thumbnails for all posts they created from pict-rs. 5. Banning a user with content removal no longer purges the community banners or icons for all communities they're the top mod of from pict-rs. 6. Purging a user no longer purges their profile avatar, banner, or images referenced in post URLs and post thumbnails for all posts they created from pict-rs. All media linked to their user account will still get deleted, which was already explicitly the case in the past. Some of the mentioned actions will still remove references to image URLs from the database, such as banning a user with content removal setting their avatar and profile banner to `NULL` in the db, which prevents them from being displayed on their profile. As this stops erasure of thumbnails, LemmyNet#5564 has been created to ensure tracking the person that triggered the creation of thumbnails, which will allow removing them like other images. The only remaining option to purge images attached to a post is now purging an individual post, which still erases the post URL and thumbnail from pict-rs entirely, including any other aliases. Purging users will remove all aliases associated with them, which will end up deleting those images entirely when there are no other alias remaining. backports LemmyNet#5565 to 0.19 LemmyNet#5565 LemmyNet@988638d

crates/api/src/site/purge/community.rs

crates/api_common/src/utils.rs

This commit removes various cases in which Lemmy would purge images from pictrs, which would in many cases not be what admins intended. pict-rs provides aliases on upload, which allows deduplicating multiple uploads of the same file in the backend, while still providing unique URLs for uploads. As Lemmy used pictrs' purge API, this meant that not only the image alias referring to removed content was deleted, all other aliases were invalidated as well. Additionally, even alias deletion would not appropriate in many of these cases, as they were lacking validation that they were exclusively used by the content they were supposed to get removed with. This implements the following changes: 1. Purging a community no longer purges the community banner or icon from pict-rs. 2. Purging a community no longer purges all images referenced in post URLs and post thumbnails within the community from pict-rs. 3. Banning a user with content removal no longer purges their profile avatar or banner from pict-rs. 4. Banning a user with content removal no longer purges images referenced in post URLs and post thumbnails for all posts they created from pict-rs. 5. Banning a user with content removal no longer purges the community banners or icons for all communities they're the top mod of from pict-rs. 6. Banning a user with content removal now deletes all media they uploaded. 7. Purging a user no longer purges their profile avatar, banner, or images referenced in post URLs and post thumbnails for all posts they created from pict-rs. All media linked to their user account will still get deleted, which was already explicitly the case in the past. 8. On an NSFW-disabled instance, receiving a post update via federation, which changes the post from not NSFW to NSFW no longer purges post URL and thumbnail from pict-rs. Some of the mentioned actions will still remove references to image URLs from the database, such as purging a community will still set its icon and banner to `NULL` in the db, but the associated images will no longer be purged from pict-rs. As this stops erasure of thumbnails, LemmyNet#5564 has been created to ensure tracking the person that triggered the creation of thumbnails, which will allow removing them like other images. The only remaining option to purge images attached to a post is now purging an individual post, which still erases the post URL and thumbnail from pict-rs entirely, including any other aliases. Purging and banning users with content removal will remove all aliases associated with them, which will end up deleting those images entirely when there are no other alias remaining. fixes LemmyNet#5560

This commit removes various cases in which Lemmy would purge images from pictrs, which would in many cases not be what admins intended. pict-rs provides aliases on upload, which allows deduplicating multiple uploads of the same file in the backend, while still providing unique URLs for uploads. As Lemmy used pictrs' purge API, this meant that not only the image alias referring to removed content was deleted, all other aliases were invalidated as well. Additionally, even alias deletion would not appropriate in many of these cases, as they were lacking validation that they were exclusively used by the content they were supposed to get removed with. This implements the following changes: 1. Purging a community no longer purges the community banner or icon from pict-rs. 2. Purging a community no longer purges all images referenced in post URLs and post thumbnails within the community from pict-rs. 3. Banning a user with content removal no longer purges their profile avatar or banner from pict-rs. 4. Banning a user with content removal no longer purges images referenced in post URLs and post thumbnails for all posts they created from pict-rs. 5. Banning a user with content removal no longer purges the community banners or icons for all communities they're the top mod of from pict-rs. 6. Banning a user with content removal now deletes all media they uploaded. 7. Purging a user no longer purges their profile avatar, banner, or images referenced in post URLs and post thumbnails for all posts they created from pict-rs. All media linked to their user account will still get deleted, which was already explicitly the case in the past. Some of the mentioned actions will still remove references to image URLs from the database, such as purging a community will still set its icon and banner to `NULL` in the db, but the associated images will no longer be purged from pict-rs. As this stops erasure of thumbnails, LemmyNet#5564 has been created to ensure tracking the person that triggered the creation of thumbnails, which will allow removing them like other images. The only remaining option to purge images attached to a post is now purging an individual post, which still erases the post URL and thumbnail from pict-rs entirely, including any other aliases. Purging and banning users with content removal will remove all aliases associated with them, which will end up deleting those images entirely when there are no other alias remaining. backports LemmyNet#5565 to 0.19 LemmyNet#5565 LemmyNet@988638d

dessalines

LGTM, thx!

dessalines · 2025-04-01T01:21:56Z

crates/apub/src/objects/post.rs

-      purge_post_images(url, thumbnail_url, context).await;
+      // TODO: Remove locally generated thumbnail if one exists, depends on
+      //       https://github.com/LemmyNet/lemmy/issues/5564 to be implemented to be able to
+      //       safely do this.


I'll comment in the linked issue.

This commit removes various cases in which Lemmy would purge images from pictrs, which would in many cases not be what admins intended. pict-rs provides aliases on upload, which allows deduplicating multiple uploads of the same file in the backend, while still providing unique URLs for uploads. As Lemmy used pictrs' purge API, this meant that not only the image alias referring to removed content was deleted, all other aliases were invalidated as well. Additionally, even alias deletion would not appropriate in many of these cases, as they were lacking validation that they were exclusively used by the content they were supposed to get removed with. This implements the following changes: 1. Purging a community no longer purges the community banner or icon from pict-rs. 2. Purging a community no longer purges all images referenced in post URLs and post thumbnails within the community from pict-rs. 3. Banning a user with content removal no longer purges their profile avatar or banner from pict-rs. 4. Banning a user with content removal no longer purges images referenced in post URLs and post thumbnails for all posts they created from pict-rs. 5. Banning a user with content removal no longer purges the community banners or icons for all communities they're the top mod of from pict-rs. 6. Banning a user with content removal now deletes all media they uploaded. 7. Purging a user no longer purges their profile avatar, banner, or images referenced in post URLs and post thumbnails for all posts they created from pict-rs. All media linked to their user account will still get deleted, which was already explicitly the case in the past. Some of the mentioned actions will still remove references to image URLs from the database, such as purging a community will still set its icon and banner to `NULL` in the db, but the associated images will no longer be purged from pict-rs. As this stops erasure of thumbnails, LemmyNet#5564 has been created to ensure tracking the person that triggered the creation of thumbnails, which will allow removing them like other images. The only remaining option to purge images attached to a post is now purging an individual post, which still erases the post URL and thumbnail from pict-rs entirely, including any other aliases. Purging and banning users with content removal will remove all aliases associated with them, which will end up deleting those images entirely when there are no other alias remaining. backports LemmyNet#5565 to 0.19 LemmyNet#5565 LemmyNet@988638d

This change was implemented after the original PR was created to restore functionality to delete user media on bans with content removal. Unlike the original implementation, this now only deletes user media rather than purging their images from pict-rs, and it applies to all uploads associated with the user rather than iterating over media references in user avatar, user banner, as well as post URLs and post thumbnails for all posts created by the banned user. backports remaining part of LemmyNet#5565 to 0.19 LemmyNet#5565 LemmyNet@b892f3d

This commit removes various cases in which Lemmy would purge images from pictrs, which would in many cases not be what admins intended. pict-rs provides aliases on upload, which allows deduplicating multiple uploads of the same file in the backend, while still providing unique URLs for uploads. As Lemmy used pictrs' purge API, this meant that not only the image alias referring to removed content was deleted, all other aliases were invalidated as well. Additionally, even alias deletion would not appropriate in many of these cases, as they were lacking validation that they were exclusively used by the content they were supposed to get removed with. This implements the following changes: 1. Purging a community no longer purges the community banner or icon from pict-rs. 2. Purging a community no longer purges all images referenced in post URLs and post thumbnails within the community from pict-rs. 3. Banning a user with content removal no longer purges their profile avatar or banner from pict-rs. 4. Banning a user with content removal no longer purges images referenced in post URLs and post thumbnails for all posts they created from pict-rs. 5. Banning a user with content removal no longer purges the community banners or icons for all communities they're the top mod of from pict-rs. 6. Banning a user with content removal now deletes all media they uploaded. 7. Purging a user no longer purges their profile avatar, banner, or images referenced in post URLs and post thumbnails for all posts they created from pict-rs. All media linked to their user account will still get deleted, which was already explicitly the case in the past. Some of the mentioned actions will still remove references to image URLs from the database, such as purging a community will still set its icon and banner to `NULL` in the db, but the associated images will no longer be purged from pict-rs. As this stops erasure of thumbnails, #5564 has been created to ensure tracking the person that triggered the creation of thumbnails, which will allow removing them like other images. The only remaining option to purge images attached to a post is now purging an individual post, which still erases the post URL and thumbnail from pict-rs entirely, including any other aliases. Purging and banning users with content removal will remove all aliases associated with them, which will end up deleting those images entirely when there are no other alias remaining. backports #5565 to 0.19 #5565 988638d

Nothing4You requested review from Nutomic, SleeplessOne1917, dessalines, dullbananas and phiresky as code owners March 29, 2025 04:17

Nothing4You force-pushed the pictrs-purges branch from 8b62dc9 to 988638d Compare March 29, 2025 04:44

Nothing4You mentioned this pull request Mar 29, 2025

[0.19] Remove unnecessary pictrs purge calls #5566

Merged

flamingos-cant added a commit to flamingos-cant/lemmy that referenced this pull request Mar 30, 2025

Remove pictrs purge from NSFW removal

281c7e5

Also add the changes to from LemmyNet#5565 related to handling NSFW posts from federation.

Nothing4You force-pushed the pictrs-purges branch from 988638d to e2565cf Compare March 31, 2025 08:09

Nothing4You force-pushed the pictrs-purges branch from e2565cf to cf9ddd5 Compare March 31, 2025 11:40

Nutomic reviewed Mar 31, 2025

View reviewed changes

crates/api/src/site/purge/community.rs Show resolved Hide resolved

Nutomic reviewed Mar 31, 2025

View reviewed changes

crates/db_schema/src/impls/post.rs Show resolved Hide resolved

Nutomic reviewed Mar 31, 2025

View reviewed changes

crates/api_common/src/utils.rs Show resolved Hide resolved

dessalines reviewed Mar 31, 2025

View reviewed changes

crates/api/src/site/purge/community.rs Show resolved Hide resolved

crates/api_common/src/utils.rs Show resolved Hide resolved

Nothing4You force-pushed the pictrs-purges branch from cf9ddd5 to 6e82f97 Compare March 31, 2025 21:25

dessalines approved these changes Apr 1, 2025

View reviewed changes

Nutomic approved these changes Apr 1, 2025

View reviewed changes

Nutomic merged commit b892f3d into LemmyNet:main Apr 1, 2025
1 check passed

Nothing4You deleted the pictrs-purges branch April 1, 2025 13:25

Nutomic mentioned this pull request Apr 3, 2025

[Bug]: Images uploaded and linked in post bodies are not purged but the references to them are lost #3504

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Remove unnecessary pictrs purge calls #5565

Remove unnecessary pictrs purge calls #5565

Uh oh!

Nothing4You commented Mar 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dessalines left a comment

Uh oh!

dessalines Apr 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Remove unnecessary pictrs purge calls #5565

Remove unnecessary pictrs purge calls #5565

Uh oh!

Conversation

Nothing4You commented Mar 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dessalines left a comment

Choose a reason for hiding this comment

Uh oh!

dessalines Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Nothing4You commented Mar 29, 2025 •

edited

Loading