fix: provide remote servers a way to find out about an event created during the remote join handshake by FrenchGithubUser · Pull Request #19390 · element-hq/synapse

FrenchGithubUser · 2026-01-19T11:33:57Z

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

TLDR

Use a "dummy" event to tie together forward extremities, and proactively send it to all servers in the room. This allows recently joined servers to become aware of recent events that would otherwise have "slipped through the cracks" and thus not be retrievable.

NOTE: While this does send the "dummy" event to all servers in the room, regardless of if they should care or not, at some point a new event will reference this dummy event and require it's retrieval. Since it was proactively sent, this will now not be necessary. This assists in preventing forks in the DAG

Alternatives

Unlike famedly/synapse#51 which 'pushes' the missing event directly, this causes the event to be 'pulled' by referencing it as a prev_event of a dummy event. Since the 'dummy event' does not get passed into the client, it is effectively invisible.

Draw-backs of famedly/synapse#51 meant it was not always certain if the 'pushed event' would show up in /sync or in /messages, but usually was in /sync. This method always has the 'missing event' show up in /messages, which I feel is more technically correct as that event was(albeit just barely) created before the 'join event' is persisted.

The Process

The order of events:

make_join from remote server, response sent
Message A sent from local server
send_join from remote server, response from local server. Message A is not in this(as it is not state and is not referenced in any events that are included). Join event is persisted on local server.
Local server realizes there are two forward extremities just after persisting the join event.
A. Creates a org.matrix.dummy_event that has prev_events containing both the join and message A.
B. Sends this dummy event to all servers in the room.
Remote server receives the dummy event via it's /send endpoint, saves it in a queue until the partial state join begins syncing additional room state

CLAassistant · 2026-01-19T11:34:04Z

All committers have signed the CLA.

FrenchGithubUser · 2026-01-19T11:48:10Z

I am submitting this PR as an employee of Famedly, who has signed the corporate CLA, and used my company email in the commit.

anoadragon453

Hi @FrenchGithubUser. As far as the Element Backend team is aware, Famedly has not yet signed the CCLA. However this is apparently currently in progress.

Holding off on review until that's resolved. Regardless, thank you for submitting your work upstream!

ara4n · 2026-02-12T00:47:38Z

famedly has now signed the ccla :)

sandhose · 2026-02-19T14:37:11Z

@FrenchGithubUser I think to allow the CLA bot to let you through, your membership to the famedly organisation must be public

If you don't want that to be the case, I can add you specifically to the list of allowed users, but making the org membership public is easier for us :)

FrenchGithubUser · 2026-02-19T14:56:25Z

@sandhose I just updated the membership, should be public now! I didn't know this visibility could be changed :)

anoadragon453 · 2026-02-27T14:39:19Z

@FrenchGithubUser could you update the branch (just pull from develop)? It seems the CI is not running, and I don't have permission to push to your branch to do it for you.

during the remote join handshake

FrenchGithubUser · 2026-02-27T14:41:18Z

@anoadragon453 done

anoadragon453

Interesting solution - thanks for sending it upstream.

I'd really like to see a Complement test for this if possible, so we can verify that this fixes the problem. I think you'd just need to send an event into the room between the /make_join and /send_join requests.

Existing federated room join tests: https://github.com/matrix-org/complement/blob/main/tests/federation_room_join_test.go

anoadragon453 · 2026-02-27T15:13:31Z

synapse/federation/federation_server.py

        event, context = await self._on_send_membership_event(
            origin, content, Membership.JOIN, room_id
        )
+        # Collect this now, the internal metadata of event(which should have it) doesn't


This comment is quite difficult to read.

And why not query this close to where it's used, below?

synapse/federation/federation_server.py

anoadragon453 · 2026-02-27T15:52:08Z

synapse/handlers/message.py

+        if not dummy_event_sent:
+            # Did not find a valid user in the room, so remove from future attempts
+            # Exclusion is time limited, so the room will be rechecked in the future
+            # dependent on _DUMMY_EVENT_ROOM_EXCLUSION_EXPIRY
+            logger.info(
+                "Failed to send dummy event into room %s. Will exclude it from "
+                "future attempts until cache expires",
+                room_id,
+            )
+            # This mapping is room_id -> time of last attempt(in ms)
+            self._rooms_to_exclude_from_dummy_event_insertion[room_id] = (
+                self.clock.time_msec()
+            )


Is it worth waiting an trying again if the room will move on in the meantime, and new events will likely be sent?

synapse/handlers/message.py

FrenchGithubUser · 2026-03-04T09:49:03Z

@anoadragon453 where would you like the complement test to be, in the complement repo or the synapse repo?

anoadragon453 · 2026-03-04T10:59:33Z

@FrenchGithubUser as I believe this would be applicable to any homeserver implementation, the Complement repo itself would be best. Thanks!

Copilot

Pull request overview

This PR aims to prevent events created during the /make_join → /send_join handshake from being “missed” by the joining remote server by injecting a dummy event which references the current forward extremities and is proactively federated.

Changes:

Add a new post-remote-join path to create/send a dummy event with internal_metadata.proactively_send=True.
On /send_join, detect multiple forward extremities and trigger dummy event injection.
Add a bugfix changelog entry describing the behavior change.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
`synapse/handlers/message.py`	Adds `_send_dummy_event_after_room_join` and makes proactive-send configurable on dummy events.
`synapse/federation/federation_server.py`	After handling `/send_join`, checks forward extremities and triggers dummy event injection.
`changelog.d/19390.bugfix`	Documents the bugfix for handshake-created events being missed by newly joined servers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

synapse/federation/federation_server.py

+        forward_extremities = await self.store._get_forward_extremeties_for_room(
+            room_id, stream_ordering_of_join.get_max_stream_pos()
+        )
+
+        if len(forward_extremities) > 1:
+            # The likelihood of this being used is extremely low, thus only build the handler
+            # when necessary.
+            _creation_handler = self.hs.get_event_creation_handler()
+            await _creation_handler._send_dummy_event_after_room_join(room_id)


synapse/federation/federation_server.py

+        if len(forward_extremities) > 1:
+            # The likelihood of this being used is extremely low, thus only build the handler
+            # when necessary.
+            _creation_handler = self.hs.get_event_creation_handler()
+            await _creation_handler._send_dummy_event_after_room_join(room_id)
+


synapse/federation/federation_server.py

        event, context = await self._on_send_membership_event(
            origin, content, Membership.JOIN, room_id
        )
+        # Collect this now, the internal metadata of event(which should have it) doesn't


synapse/handlers/message.py

+        This should only be triggered when handling a remote join while there was
+        events sent during the make_join/send_join handshake. The joining
+        homeserver would otherwise not immediately know to backfill this event,
+        and would "miss it".


synapse/federation/federation_server.py

+        # Check the forward extremities for the room here. If there is more than one, it
+        # is likely that another event was created in the room during the
+        # make_join/send_join handshake. The joining server is likely to thus miss this event
+        # until a second event is created when references it - which could be some time.
+        # In that case, we proactively send a dummy extensible event that ties these
+        # forward extremities together. The remote server will then attempt to backfill
+        # the missing event on its own.
+        #
+        # By not sending the 'missing event' directly, but instead having the joining
+        # homeserver backfill it, the stream ordering for the missing event will be
+        # "before" the join (which is what we expect).
+
+        forward_extremities = await self.store._get_forward_extremeties_for_room(
+            room_id, stream_ordering_of_join.get_max_stream_pos()
+        )
+
+        if len(forward_extremities) > 1:
+            # The likelihood of this being used is extremely low, thus only build the handler
+            # when necessary.
+            _creation_handler = self.hs.get_event_creation_handler()
+            await _creation_handler._send_dummy_event_after_room_join(room_id)
+


changelog.d/19390.bugfix

@@ -0,0 +1 @@
+Provide remote servers a way to find out about an event created during the remote join handshake. Contributed by @FrenchGithubUser and @jason-famedly @ Famedly.


synapse/federation/federation_server.py

+        # Collect this now, the internal metadata of event(which should have it) doesn't
+        stream_ordering_of_join = (
+            await self.store.get_current_room_stream_token_for_room_id(room_id)


FrenchGithubUser force-pushed the join-race-condition branch from e29abe1 to 688aca5 Compare January 19, 2026 11:35

FrenchGithubUser marked this pull request as ready for review January 19, 2026 11:36

FrenchGithubUser requested a review from a team as a code owner January 19, 2026 11:36

FrenchGithubUser force-pushed the join-race-condition branch from 688aca5 to 9483291 Compare January 20, 2026 12:41

MadLittleMods mentioned this pull request Jan 20, 2026

fix: Cap WorkerLock timeout intervals to 15 minutes #19394

Open

3 tasks

anoadragon453 requested changes Feb 4, 2026

View reviewed changes

anoadragon453 mentioned this pull request Feb 5, 2026

fix: Avoid a M_FORBIDDEN response when a user tries to erase their account and profile updates are disabled #19398

Open

3 tasks

MadLittleMods requested a review from a team February 12, 2026 00:48

fix: provide remote servers a way to find out about an event created

952ebb5

during the remote join handshake

FrenchGithubUser force-pushed the join-race-condition branch from 9483291 to 952ebb5 Compare February 27, 2026 14:40

MadLittleMods added A-Federated-Join A-Join labels Feb 27, 2026

anoadragon453 requested changes Feb 27, 2026

View reviewed changes

address some comments

22cc6ac

FrenchGithubUser mentioned this pull request Mar 4, 2026

test: events sent during remote join handshake should not be lost matrix-org/complement#847

Open

1 task

MadLittleMods mentioned this pull request Mar 5, 2026

First message sent in a federated server room is not visible to a newly joined user #17560

Open

anoadragon453 requested a review from Copilot March 18, 2026 16:27

Copilot started reviewing on behalf of anoadragon453 March 18, 2026 16:27 View session

Copilot AI reviewed Mar 18, 2026

View reviewed changes

		@@ -0,0 +1 @@
		Provide remote servers a way to find out about an event created during the remote join handshake. Contributed by @FrenchGithubUser and @jason-famedly @ Famedly.

Conversation

FrenchGithubUser commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

TLDR

Alternatives

The Process

Uh oh!

CLAassistant commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrenchGithubUser commented Jan 19, 2026

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

ara4n commented Feb 12, 2026

Uh oh!

sandhose commented Feb 19, 2026

Uh oh!

FrenchGithubUser commented Feb 19, 2026

Uh oh!

anoadragon453 commented Feb 27, 2026

Uh oh!

FrenchGithubUser commented Feb 27, 2026

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

anoadragon453 Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anoadragon453 Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FrenchGithubUser commented Mar 4, 2026

Uh oh!

anoadragon453 commented Mar 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

FrenchGithubUser commented Jan 19, 2026 •

edited

Loading

CLAassistant commented Jan 19, 2026 •

edited

Loading