Implement synapse issue #16751: Treat local_media_directory as optional storage provider by drallgood · Pull Request #19204 · element-hq/synapse

drallgood · 2025-11-19T12:57:34Z

This allows off-site media storage without local cache by making the local media directory behave like any other MediaStorageProvider. When no local directory is configured, media is stored directly to remote providers only, with temporary files used for thumbnail generation when needed.

fixes #16751

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

… as optional storage provider This allows off-site media storage without local cache by making the local media directory behave like any other MediaStorageProvider. When no local directory is configured, media is stored directly to remote providers only, with temporary files used for thumbnail generation when needed.

CLAassistant · 2025-11-19T12:57:43Z

All committers have signed the CLA.

… as optional storage provider This allows off-site media storage without local cache by making the local media directory behave like any other MediaStorageProvider. When no local directory is configured, media is stored directly to remote providers only, with temporary files used for thumbnail generation when needed. Also fixes URL cache thumbnails to be served directly from local filesystem instead of being incorrectly routed through storage providers.

These were auto-formatted changes that weren't intended to be committed.

Add type ignore comment for os.path.join call since local_provider check guarantees local_media_directory is not None.

anoadragon453

Thank you for your contribution. Several questions below.

From the linked issues, I think broadly what we're aiming for is to satisfy the use-case where media isn't stored on disk at all, and is only stored on (typically) remote providers.

"nit" below means a non-blocking change. Typically related to code style or readability.

synapse/media/media_repository.py

synapse/media/media_storage.py

Address code review feedback: - Add enable_local_media_storage config option to disable local media store - Refactor MediaStorage.__init__ to accept local_provider as optional FileStorageProviderBackend argument - Rename fname to media_filepath and eliminate dirname variable - Deduplicate spam check code in store_into_file - Delete temp files immediately after upload to storage providers - Fix fetch_media to check local_provider first - Remove unnecessary list() wrapper - Revert storage_providers type to Sequence[StorageProvider] - Update tests to use new MediaStorage constructor signature

When no local provider is configured, temp files are now automatically cleaned up after use. This addresses the reviewer feedback about temp files building up on disk over time. The method now yields the file path within a context manager, and callers must use `async with` to ensure proper cleanup.

drallgood · 2025-12-03T21:01:45Z

@anoadragon453 thanks for the review. I reworked the suggested parts and found another area (the thumbnail generation) that needed fixing.

Hope this is better? 😄

Add enable_local_media_storage to the JSON schema and regenerate the configuration documentation so the new flag appears under the Media Store section.

anoadragon453

Looking mostly good now! Thanks for making those changes.

Couple more things, plus the following unit tests are currently failing:

tests.replication.test_multi_media_repo.AuthenticatedMediaRepoShardTestCase.test_download_simple_file_race
tests.replication.test_multi_media_repo.MediaRepoShardTestCase.test_download_simple_file_race

These tests assume that the files we fetch remain on disk, and this assumption should continue to hold as enable_local_media_storage is True in tests.

It'd also be good to see some tests with enable_local_media_storage set to False, so we can check the happy path works, and that files aren't pooling up on disk.

Note that you can use the @override_config decorator on tests to dynamically modify homeserver configuration for just that test case.

synapse/media/media_storage.py

synapse/media/thumbnailer.py

- Move database insert outside store_into_file context manager to ensure proper cleanup of duplicate files on constraint violations - Add explicit file cleanup on IntegrityError when concurrent downloads create duplicate files - Refactor duplicate exception handling into _store_remote_media_with_cleanup helper method - Add tests for enable_local_media_storage=False to verify no local file accumulation when local storage is disabled - Remove unnecessary ensure_media_is_in_local_cache call in thumbnailer.py Fixes race condition where two workers concurrently downloading the same media would result in two files being stored instead of one.

drallgood · 2026-01-02T13:23:15Z

I think I fixed all tests and also added the suggested tests. It's a bit tricky to get proper tests with external providers up and running so I went with a compromise.
@anoadragon453

anoadragon453

Just a few small changes, and I think this is good to go! Thanks for your patience with the back and forth.

tests/replication/test_multi_media_repo.py

schema/synapse-config.schema.yaml

changelog.d/19204.feature

schema/synapse-config.schema.yaml

Co-authored-by: Andrew Morgan <[email protected]>

When enable_local_media_storage is set to false and no media_storage_providers are configured, all media requests will return 404 errors as there will be no storage backend available. This addresses reviewer feedback to document the expected behavior in this configuration scenario.

When enable_local_media_storage is disabled and no media_storage_providers are configured, log a warning at startup to inform administrators that all media requests will return 404 errors due to the lack of storage backend.

anoadragon453

Excited to see this land. Thanks for your patience with the review process!

drallgood · 2026-01-06T21:32:59Z

Excited to see this land. Thanks for your patience with the review process!

Yay. No worries. We have to get this right!
Also thanks for regenerating the docs. I forgot to do that.

Ref: - #4882 - element-hq/synapse#19204 - https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#enable_local_media_storage We default it to `true`, keeping up with upstream and the old behavior. s3-storage-provider users may set `matrix_synapse_enable_local_media_storage` to `false` to disable local file caching. This likely comes at the expense of some performance. For matrix-media-repo users, it likely doesn't matter what this is set to, as for a matrix-media-repo setup, all media-related API endpoints are captured and forwarded to matrix-media-repo (before reaching Synapse).

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [element-hq/synapse](https://github.com/element-hq/synapse) | minor | `1.145.0` → `1.146.0` | --- > ⚠️ **Warning** > > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>element-hq/synapse (element-hq/synapse)</summary> ### [`v1.146.0`](https://github.com/element-hq/synapse/releases/tag/v1.146.0) [Compare Source](element-hq/synapse@v1.145.0...v1.146.0rc1) ### Synapse 1.146.0 (2026-01-27) No significant changes since 1.146.0rc1. #### Deprecations and Removals - [MSC2697](matrix-org/matrix-spec-proposals#2697) (Dehydrated devices) has been removed, as the MSC is closed. Developers should migrate to [MSC3814](matrix-org/matrix-spec-proposals#3814). ([#19346](element-hq/synapse#19346)) - Support for Ubuntu 25.04 (Plucky Puffin) has been dropped. Synapse no longer builds debian packages for Ubuntu 25.04. ### Synapse 1.146.0rc1 (2026-01-20) #### Features - Add a new config option [`enable_local_media_storage`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#enable_local_media_storage) which controls whether media is additionally stored locally when using configured `media_storage_providers`. Setting this to `false` allows off-site media storage without a local cache. Contributed by Patrice Brend'amour [@dr](https://github.com/dr).allgood. ([#19204](element-hq/synapse#19204)) - Stabilise support for [MSC4312](matrix-org/matrix-spec-proposals#4312 `m.oauth` User-Interactive Auth stage for resetting cross-signing identity with the OAuth 2.0 API. The old, unstable name (`org.matrix.cross_signing_reset`) is now deprecated and will be removed in a future release. ([#19273](element-hq/synapse#19273)) - Refactor Grafana dashboard to use `server_name` label (instead of `instance`). ([#19337](element-hq/synapse#19337)) #### Bugfixes - Fix joining a restricted v12 room locally when no local room creator is present but local users with sufficient power levels are. Contributed by [@nexy7574](https://github.com/nexy7574). ([#19321](element-hq/synapse#19321)) - Fixed parallel calls to `/_matrix/media/v1/create` being ratelimited for appservices even if `rate_limited: false` was set in the registration. Contributed by [@tulir](https://github.com/tulir) @ Beeper. ([#19335](element-hq/synapse#19335)) - Fix a bug introduced in 1.61.0 where a user's membership in a room was accidentally ignored when considering access to historical state events in rooms with the "shared" history visibility. Contributed by Lukas Tautz. ([#19353](element-hq/synapse#19353)) - [MSC4140](matrix-org/matrix-spec-proposals#4140): Store the JSON content of scheduled delayed events as text instead of a byte array. This fixes the inability to schedule a delayed event with non-ASCII characters in its content. ([#19360](element-hq/synapse#19360)) - Always rollback database transactions when retrying (avoid orphaned connections). ([#19372](element-hq/synapse#19372)) - Fix `InFlightGauge` typing to allow upgrading to `prometheus_client` 0.24. ([#19379](element-hq/synapse#19379)) #### Updates to the Docker image - Add [Prometheus HTTP service discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) endpoint for easy discovery of all workers when using the `docker/Dockerfile-workers` image (see the [*Metrics* section of our Docker testing docs](docker/README-testing.md#metrics)). ([#19336](element-hq/synapse#19336)) #### Improved Documentation - Remove docs on legacy metric names (no longer in the codebase since 2022-12-06). ([#19341](element-hq/synapse#19341)) - Clarify how the estimated value of room complexity is calculated internally. ([#19384](element-hq/synapse#19384)) #### Internal Changes - Add an internal `cancel_task` API to the task scheduler. ([#19310](element-hq/synapse#19310)) - Tweak docstrings and signatures of `auth_types_for_event` and `get_catchup_room_event_ids`. ([#19320](element-hq/synapse#19320)) - Replace usage of deprecated `assertEquals` with `assertEqual` in unit test code. ([#19345](element-hq/synapse#19345)) - Drop support for Ubuntu 25.04 'Plucky Puffin', add support for Ubuntu 25.10 'Questing Quokka'. ([#19348](element-hq/synapse#19348)) - Revert "Add an Admin API endpoint for listing quarantined media ([#19268](element-hq/synapse#19268))". ([#19351](element-hq/synapse#19351)) - Bump `mdbook` from 0.4.17 to 0.5.2 and remove our custom table-of-contents plugin in favour of the new default functionality. ([#19356](element-hq/synapse#19356)) - Replace deprecated usage of PyGitHub's `GitRelease.title` with `.name` in release script. ([#19358](element-hq/synapse#19358)) - Update the Element logo in Synapse's README to be an absolute URL, allowing it to render on other sites (such as PyPI). ([#19368](element-hq/synapse#19368)) - Apply minor tweaks to v1.145.0 changelog. ([#19376](element-hq/synapse#19376)) - Update Grafana dashboard syntax to use the latest from importing/exporting with Grafana 12.3.1. ([#19381](element-hq/synapse#19381)) - Warn about skipping reactor metrics when using unknown reactor type. ([#19383](element-hq/synapse#19383)) - Add support for reactor metrics with the `ProxiedReactor` used in worker Complement tests. ([#19385](element-hq/synapse#19385)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://gitea.alexlebens.dev/alexlebens/infrastructure/pulls/3533 Co-authored-by: Renovate Bot <[email protected]> Co-committed-by: Renovate Bot <[email protected]>

# Famedly Synapse Release v1.146.0_1 depends on: famedly/complement#10 ## Famedly additions for v1.146.0_1 - feat: trigger CI actions (that are triggered on PRs) in merge queue (FrenchGithubUser) ### Notes for Famedly: #### Deprecations and Removals - matrix-org/matrix-spec-proposals#2697 (Dehydrated devices) has been removed, as the MSC is closed. Developers should migrate to matrix-org/matrix-spec-proposals#3814. (element-hq/synapse#19346) - Support for Ubuntu 25.04 (Plucky Puffin) has been dropped. Synapse no longer builds debian packages for Ubuntu 25.04. #### Updates to the Docker image - Add [Prometheus HTTP service discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) endpoint for easy discovery of all workers when using the docker/Dockerfile-workers image (see the [Metrics section of our Docker testing docs](https://github.com/famedly/synapse/pull/docker/README-testing.md#metrics)). (element-hq/synapse#19336) #### Features - Add a new config option [enable_local_media_storage](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#enable_local_media_storage) which controls whether media is additionally stored locally when using configured media_storage_providers. Setting this to false allows off-site media storage without a local cache. Contributed by Patrice Brend'amour @dr.allgood. (element-hq/synapse#19204) - Stabilise support for matrix-org/matrix-spec-proposals#4312 m.oauth User-Interactive Auth stage for resetting cross-signing identity with the OAuth 2.0 API. The old, unstable name (org.matrix.cross_signing_reset) is now deprecated and will be removed in a future release. (element-hq/synapse#19273) - Refactor Grafana dashboard to use server_name label (instead of instance). (element-hq/synapse#19337)

drallgood added 2 commits November 19, 2025 23:20

Revert lint.sh formatting changes

8eed314

These were auto-formatted changes that weren't intended to be committed.

drallgood changed the title ~~WIP: Implement synapse issue #16751: Treat local_media_directory as optional storage provider~~ Implement synapse issue #16751: Treat local_media_directory as optional storage provider Nov 19, 2025

drallgood marked this pull request as ready for review November 19, 2025 22:26

drallgood requested a review from a team as a code owner November 19, 2025 22:26

Fix mypy error for local_media_directory type annotation

b6ffef6

Add type ignore comment for os.path.join call since local_provider check guarantees local_media_directory is not None.

MadLittleMods added the A-Media-Repository label Nov 24, 2025

Merge branch 'develop' into implement-issue-16751-local-media-provider

ccc047d

anoadragon453 requested changes Nov 28, 2025

View reviewed changes

drallgood added 2 commits December 3, 2025 21:34

drallgood added 4 commits December 3, 2025 22:08

Document enable_local_media_storage config option

94c4358

Add enable_local_media_storage to the JSON schema and regenerate the configuration documentation so the new flag appears under the Media Store section.

Merge branch 'develop' into implement-issue-16751-local-media-provider

f63c91a

Fix lint formatting in media_repository

99eb251

Merge branch 'develop' into implement-issue-16751-local-media-provider

af958d3

anoadragon453 self-requested a review December 11, 2025 16:37

anoadragon453 requested changes Dec 22, 2025

View reviewed changes

synapse/media/media_storage.py Outdated Show resolved Hide resolved

synapse/media/thumbnailer.py Outdated Show resolved Hide resolved

Merge branch 'develop' into implement-issue-16751-local-media-provider

c957ae9

anoadragon453 requested changes Jan 5, 2026

View reviewed changes

tests/replication/test_multi_media_repo.py Outdated Show resolved Hide resolved

schema/synapse-config.schema.yaml Outdated Show resolved Hide resolved

changelog.d/19204.feature Outdated Show resolved Hide resolved

schema/synapse-config.schema.yaml Show resolved Hide resolved

drallgood and others added 4 commits January 5, 2026 13:33

Update tests/replication/test_multi_media_repo.py

635c4cc

Co-authored-by: Andrew Morgan <[email protected]>

Apply suggestions from code review

8992d11

Co-authored-by: Andrew Morgan <[email protected]>

Merge branch 'develop' into implement-issue-16751-local-media-provider

fedda30

drallgood requested a review from anoadragon453 January 5, 2026 12:40

Add warning log when no media storage backends are configured

96b6703

When enable_local_media_storage is disabled and no media_storage_providers are configured, log a warning at startup to inform administrators that all media requests will return 404 errors due to the lack of storage backend.

Regenerate config documentation

9459809

anoadragon453 approved these changes Jan 6, 2026

View reviewed changes

anoadragon453 enabled auto-merge (squash) January 6, 2026 17:23

anoadragon453 disabled auto-merge January 6, 2026 23:29

anoadragon453 merged commit a094d92 into element-hq:develop Jan 6, 2026
43 of 45 checks passed

drallgood deleted the implement-issue-16751-local-media-provider branch January 7, 2026 00:19

This was referenced Feb 5, 2026

Famedly release/v1.146 famedly/synapse#232

Closed

Famedly release/v1.146 famedly/synapse#236

Merged

Conversation

drallgood commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

CLAassistant commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drallgood commented Dec 3, 2025

Uh oh!

anoadragon453 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

drallgood commented Jan 2, 2026

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

drallgood commented Jan 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

drallgood commented Nov 19, 2025 •

edited

Loading

CLAassistant commented Nov 19, 2025 •

edited

Loading

anoadragon453 left a comment •

edited

Loading