MSC2314: A Method to Backfill Room State#2314
Conversation
could you update this if it's no longer correct? |
done, reflecting the current PR |
|
|
||
| If a server experiences data loss, it is difficult for it to recover membership in federation without having a user be reinvited, as backfill alone can not easily retrieve the data that is required to operate in a room (namely the current state and auth chain). | ||
|
|
||
| This MSC introduces S2S APIs to provide a given room's auth chain and current state events, provided the requesting server is in the room specified. |
There was a problem hiding this comment.
How do you see this API actually being used? The "current state" isn't really a concept that is used much in s2s, as if the server wants to start participating in the room it needs a set of most recent events and their state (from which the current state can be derived).
There was a problem hiding this comment.
Yes, it's asking another server for it's current state, in a situation where it knows it's joined to the room but is missing state (data loss).
There was a problem hiding this comment.
But how does it actually use the current state? I don't think the current state is useful in rehydrating a room for example, as you need the extremities and the state at each of them.
There was a problem hiding this comment.
hrm, it seems I missed this thread previously. @erikjohnston raises a good point - it is very hard to see how this API could be usefully used.
If we can't figure out a practical use for this change, we should instead reject it, and back out the changes in Synapse and Sytest.
@erikjohnston any further thoughts here?
There was a problem hiding this comment.
It does feel like it should be useful, but I can't really see how. If this is mainly useful for rescuing a server that has suffered catastrophic memory lost then I think I'd like to see a step-by-step write up of the process of recovering from that, including any new APIs that need to be added.
Having a one stop "bootstrap this room for me" API might be useful, which returns the extremities, auth chain and state at each event (probably with some clever de-duplication there)
There was a problem hiding this comment.
right then. In that case, I'm going to close this MSC, and back out the changes in sytest and synapse.
|
|
||
| Excessively large rooms may cause performance problems for servers implementing this API (including in its v1 incarnation). Discretionary rate limiting of this API may be required. | ||
|
|
||
| ## Security considerations |
There was a problem hiding this comment.
servers might also have to worry about malicious attempts to get the server to join rooms. A server that really wants you to join a room could send the server an event over /send, therefore confusing the server into thinking it might be in the room. The victim server would then use this new API, find out some information about the room (possibly a falsified join event) and push the room through as joined. In theory at some point a signature check would fail, but another workaround could be to ask a couple servers in the room (if possible) to verify the state returned by a server.
turt2live
left a comment
There was a problem hiding this comment.
seems generally sensible to me
|
So by my understanding, both Synapse and Construct implement this MSC as is on the stable endpoint. |
|
It has come to my attention (thanks to @jevolk) that it would be a good idea for |
|
I'm not sure what happened to this, but Synapse's sytest blacklist is written as if this is formally merged into the spec: |
|
This landed in synapse (matrix-org/synapse#6176), hence the change to the sytest blacklist. We should either rip it out of synapse, or land it in the spec. I think the latter is doable, hence: @mscbot fcp merge |
|
This FCP proposal has been cancelled by #2314 (comment). Team member @mscbot has proposed to merge this. The next step is review by the rest of the tagged people: Once at least 75% of reviewers approve (and there are no outstanding concerns), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up! See this document for information about what commands tagged team members can give me. |
anoadragon453
left a comment
There was a problem hiding this comment.
One small q, the rest looks good to me.
|
closing this as it doesn't seem to have a real usecase, per #2314 (comment). |
|
@mscbot fcp cancel |
Synapse 1.58.0rc1 (2022-04-26) ============================== As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](matrix-org/synapse#11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61. Features -------- - Implement [MSC3383](matrix-org/matrix-spec-proposals#3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](matrix-org/synapse#11398)) - Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](matrix-org/synapse#11537)) - Enable processing of device list updates asynchronously. ([\#12365](matrix-org/synapse#12365), [\#12465](matrix-org/synapse#12465)) - Implement [MSC2815](matrix-org/matrix-spec-proposals#2815) to allow room moderators to view redacted event content. Contributed by @tulir. ([\#12427](matrix-org/synapse#12427)) - Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](matrix-org/synapse#12543)) Bugfixes -------- - Prevent a sync request from removing a user's busy presence status. ([\#12213](matrix-org/synapse#12213)) - Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper. ([\#12319](matrix-org/synapse#12319)) - Fix a long-standing bug which incorrectly caused `GET /_matrix/client/v3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original. ([\#12476](matrix-org/synapse#12476)) - Fix a bug introduced in Synapse 1.27.0 where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1, no matter how many extremities were deleted. ([\#12496](matrix-org/synapse#12496)) - Fix a long-standing bug where the image thumbnails embedded into email notifications were broken. ([\#12510](matrix-org/synapse#12510)) - Fix a bug in the implementation of [MSC3202](matrix-org/matrix-spec-proposals#3202) where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`. ([\#12520](matrix-org/synapse#12520)) - Fix a bug introduced in Synapse 0.99.3 which could cause Synapse to consume large amounts of RAM when back-paginating in a large room. ([\#12522](matrix-org/synapse#12522)) Improved Documentation ---------------------- - Fix rendering of the documentation site when using the 'print' feature. ([\#12340](matrix-org/synapse#12340)) - Add a manual documenting config file options. ([\#12368](matrix-org/synapse#12368), [\#12527](matrix-org/synapse#12527)) - Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental. ([\#12451](matrix-org/synapse#12451)) - Update worker documentation and replace old `federation_reader` with `generic_worker`. ([\#12457](matrix-org/synapse#12457)) - Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](matrix-org/synapse#12475)) - Add some example configurations for workers and update architectural diagram. ([\#12492](matrix-org/synapse#12492)) - Fix a broken link in `README.rst`. ([\#12495](matrix-org/synapse#12495)) - Add HAProxy delegation example with CORS headers to docs. ([\#12501](matrix-org/synapse#12501)) - Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari. ([\#12533](matrix-org/synapse#12533)) Deprecations and Removals ------------------------- - The groups/communities feature in Synapse is now disabled by default. ([\#12344](matrix-org/synapse#12344)) - Remove unstable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#12382](matrix-org/synapse#12382)) Internal Changes ---------------- - Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join. ([\#12394](matrix-org/synapse#12394)) - Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive. ([\#12399](matrix-org/synapse#12399)) - Remove an unstable identifier from [MSC3083](matrix-org/matrix-spec-proposals#3083). ([\#12395](matrix-org/synapse#12395)) - Run CI in the locked [Poetry](https://python-poetry.org/) environment, and remove corresponding `tox` jobs. ([\#12425](matrix-org/synapse#12425), [\#12434](matrix-org/synapse#12434), [\#12438](matrix-org/synapse#12438), [\#12441](matrix-org/synapse#12441), [\#12449](matrix-org/synapse#12449), [\#12478](matrix-org/synapse#12478), [\#12514](matrix-org/synapse#12514), [\#12472](matrix-org/synapse#12472)) - Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current iteration of [MSC2666](matrix-org/matrix-spec-proposals#2666). ([\#12445](matrix-org/synapse#12445)) - Fix typo in the release script help string. ([\#12450](matrix-org/synapse#12450)) - Fix a minor typo in the Debian changelogs generated by the release script. ([\#12497](matrix-org/synapse#12497)) - Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development. ([\#12455](matrix-org/synapse#12455)) - Limit length of `device_id` to less than 512 characters. ([\#12454](matrix-org/synapse#12454)) - Dockerfile-workers: reduce the amount we install in the image. ([\#12464](matrix-org/synapse#12464)) - Dockerfile-workers: give the master its own log config. ([\#12466](matrix-org/synapse#12466)) - complement-synapse-workers: factor out separate entry point script. ([\#12467](matrix-org/synapse#12467)) - Back out experimental implementation of [MSC2314](matrix-org/matrix-spec-proposals#2314). ([\#12474](matrix-org/synapse#12474)) - Fix grammatical error in federation error response when the room version of a room is unknown. ([\#12483](matrix-org/synapse#12483)) - Remove unnecessary configuration overrides in tests. ([\#12511](matrix-org/synapse#12511)) - Refactor the relations code for clarity. ([\#12519](matrix-org/synapse#12519)) - Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`. ([\#12528](matrix-org/synapse#12528)) - Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s. ([\#12468](matrix-org/synapse#12468)) - Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db`. ([\#12529](matrix-org/synapse#12529))
Synapse 1.58.0 (2022-05-03) =========================== As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](matrix-org/synapse#11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61. No significant changes since 1.58.0rc2. Synapse 1.58.0rc2 (2022-04-26) ============================== This release candidate fixes bugs related to Synapse 1.58.0rc1's logic for handling device list updates. Bugfixes -------- - Fix a bug introduced in Synapse 1.58.0rc1 where the main process could consume excessive amounts of CPU and memory while handling sentry logging failures. ([\#12554](matrix-org/synapse#12554)) - Fix a bug introduced in Synapse 1.58.0rc1 where opentracing contexts were not correctly sent to whitelisted remote servers with device lists updates. ([\#12555](matrix-org/synapse#12555)) Internal Changes ---------------- - Reduce unnecessary work when handling remote device list updates. ([\#12557](matrix-org/synapse#12557)) Synapse 1.58.0rc1 (2022-04-26) ============================== Features -------- - Implement [MSC3383](matrix-org/matrix-spec-proposals#3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](matrix-org/synapse#11398)) - Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](matrix-org/synapse#11537)) - Enable processing of device list updates asynchronously. ([\#12365](matrix-org/synapse#12365), [\#12465](matrix-org/synapse#12465)) - Implement [MSC2815](matrix-org/matrix-spec-proposals#2815) to allow room moderators to view redacted event content. Contributed by @tulir @ Beeper. ([\#12427](matrix-org/synapse#12427)) - Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](matrix-org/synapse#12543)) Bugfixes -------- - Prevent a sync request from removing a user's busy presence status. ([\#12213](matrix-org/synapse#12213)) - Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper. ([\#12319](matrix-org/synapse#12319)) - Fix a long-standing bug which incorrectly caused `GET /_matrix/client/v3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original. ([\#12476](matrix-org/synapse#12476)) - Fix a bug introduced in Synapse 1.27.0 where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1, no matter how many extremities were deleted. ([\#12496](matrix-org/synapse#12496)) - Fix a long-standing bug where the image thumbnails embedded into email notifications were broken. ([\#12510](matrix-org/synapse#12510)) - Fix a bug in the implementation of [MSC3202](matrix-org/matrix-spec-proposals#3202) where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`. ([\#12520](matrix-org/synapse#12520)) - Fix a bug introduced in Synapse 0.99.3 which could cause Synapse to consume large amounts of RAM when back-paginating in a large room. ([\#12522](matrix-org/synapse#12522)) Improved Documentation ---------------------- - Fix rendering of the documentation site when using the 'print' feature. ([\#12340](matrix-org/synapse#12340)) - Add a manual documenting config file options. ([\#12368](matrix-org/synapse#12368), [\#12527](matrix-org/synapse#12527)) - Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental. ([\#12451](matrix-org/synapse#12451)) - Update worker documentation and replace old `federation_reader` with `generic_worker`. ([\#12457](matrix-org/synapse#12457)) - Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](matrix-org/synapse#12475)) - Add some example configurations for workers and update architectural diagram. ([\#12492](matrix-org/synapse#12492)) - Fix a broken link in `README.rst`. ([\#12495](matrix-org/synapse#12495)) - Add HAProxy delegation example with CORS headers to docs. ([\#12501](matrix-org/synapse#12501)) - Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari. ([\#12533](matrix-org/synapse#12533)) Deprecations and Removals ------------------------- - The groups/communities feature in Synapse is now disabled by default. ([\#12344](matrix-org/synapse#12344)) - Remove unstable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#12382](matrix-org/synapse#12382)) Internal Changes ---------------- - Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join. ([\#12394](matrix-org/synapse#12394)) - Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive. ([\#12399](matrix-org/synapse#12399)) - Remove an unstable identifier from [MSC3083](matrix-org/matrix-spec-proposals#3083). ([\#12395](matrix-org/synapse#12395)) - Run CI in the locked [Poetry](https://python-poetry.org/) environment, and remove corresponding `tox` jobs. ([\#12425](matrix-org/synapse#12425), [\#12434](matrix-org/synapse#12434), [\#12438](matrix-org/synapse#12438), [\#12441](matrix-org/synapse#12441), [\#12449](matrix-org/synapse#12449), [\#12478](matrix-org/synapse#12478), [\#12514](matrix-org/synapse#12514), [\#12472](matrix-org/synapse#12472)) - Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current iteration of [MSC2666](matrix-org/matrix-spec-proposals#2666). ([\#12445](matrix-org/synapse#12445)) - Fix typo in the release script help string. ([\#12450](matrix-org/synapse#12450)) - Fix a minor typo in the Debian changelogs generated by the release script. ([\#12497](matrix-org/synapse#12497)) - Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development. ([\#12455](matrix-org/synapse#12455)) - Limit length of `device_id` to less than 512 characters. ([\#12454](matrix-org/synapse#12454)) - Dockerfile-workers: reduce the amount we install in the image. ([\#12464](matrix-org/synapse#12464)) - Dockerfile-workers: give the master its own log config. ([\#12466](matrix-org/synapse#12466)) - complement-synapse-workers: factor out separate entry point script. ([\#12467](matrix-org/synapse#12467)) - Back out experimental implementation of [MSC2314](matrix-org/matrix-spec-proposals#2314). ([\#12474](matrix-org/synapse#12474)) - Fix grammatical error in federation error response when the room version of a room is unknown. ([\#12483](matrix-org/synapse#12483)) - Remove unnecessary configuration overrides in tests. ([\#12511](matrix-org/synapse#12511)) - Refactor the relations code for clarity. ([\#12519](matrix-org/synapse#12519)) - Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`. ([\#12528](matrix-org/synapse#12528)) - Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s. ([\#12468](matrix-org/synapse#12468)) - Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db`. ([\#12529](matrix-org/synapse#12529))
Rendered
This API can be tested in matrix-org/synapse#6176 .
Signed-off-by: Amber H. Brown [email protected]
FCP proposal: #2314 (comment)