MSC4242: State DAGs (CSAPI) by kegsay · Pull Request #19424 · element-hq/synapse

kegsay · 2026-02-03T09:33:39Z

This implements MSC4242: State DAGs, without support for federation.

A general overview:

It adds a new room version and new event type.
It adds a new field calculated_auth_event_ids to internal metadata.
It stores the state DAG via new state DAG edges / forward extremities tables.
It adds new auth rules as per the MSC.
It uses the new prev_state_events field instead of prev_event_ids() when doing state resolution.

Complement tests: matrix-org/complement#841

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

This implements state DAGs, without support for federation. A general overview: - It adds a new room version and new event type. - It adds a new field `calculated_auth_event_ids` to internal metadata. - It stores the state DAG via new state DAG edges / forward extremities tables. - It adds new auth rules as per the MSC. - It uses the new `prev_state_events` field instead of `prev_event_ids()` when doing state resolution.

Builds off #19424 Adds federation compatibility for state DAG rooms. Overview: - Adds extra HTTP API fields as per the MSC. - Adds methods for walking and extracting the state DAG for a room (for `/get_missing_events` and `/send_join` respectively). - Adds impl for processing the federation requests, as well as `/send`.

synapse/api/room_versions.py

synapse/events/__init__.py

synapse/api/room_versions.py

synapse/events/__init__.py

tests/storage/test_msc4242_state_dag.py

tests/test_event_auth.py

synapse/handlers/message.py

tests/storage/test_msc4242_state_dag.py

Co-authored-by: Eric Eastwood <erice@element.io>

synapse/events/__init__.py

synapse/handlers/admin.py

synapse/storage/controllers/persist_events.py

synapse/event_auth.py

Co-authored-by: Eric Eastwood <erice@element.io>

Spawning from #19424 (comment)

synapse/event_auth.py

tests/storage/test_msc4242_state_dag.py

MadLittleMods · 2026-03-17T02:40:12Z

tests/storage/test_msc4242_state_dag.py

+        second_id = self.helper.send(self.room_id, body="test2")["event_id"]
+        second_prev_state_event = self._get_prev_state_events(second_id)
+        assert len(second_prev_state_event) == 1
+        self.assertEquals(first_prev_state_event, second_prev_state_event)


Do we care about the order?

If not, might as well use self.assertIncludes(..., exact=True) and for better assertion diff messages.

We don't care about the order, but there's always only 1 element in this test so I don't think it will make the message hard to follow.

Using something like self.assertIncludes(..., exact=True) (or self.assertCountEqual(...)) aligns our assertion with what we care about (we don't care about order). People don't have to ask the question about whether we have to preserve the that part of the assertion through refactors.

MadLittleMods · 2026-03-17T02:45:09Z

tests/storage/test_msc4242_state_dag.py

+        self.assertEqual(
+            new_extrems,
+            want_new_extrems,
+            f"want_new_extrems={want_new_extrems} got={new_extrems}",
+        )


self.assertIncludes(..., exact=True) for better diff messages

I tried doing this but mypy doesn't like it.

tests/storage/test_msc4242_state_dag.py:67: error: Argument 1 to "assertIncludes" of "TestCase" has incompatible type "list[str]"; expected "AbstractSet[Never]" [arg-type] tests/storage/test_msc4242_state_dag.py:67: error: Argument 2 to "assertIncludes" of "TestCase" has incompatible type "list[str]"; expected "AbstractSet[Never]" [arg-type] tests/storage/test_msc4242_state_dag.py:83: error: Argument 1 to "assertIncludes" of "TestCase" has incompatible type "list[str]"; expected "AbstractSet[Never]" [arg-type] tests/storage/test_msc4242_state_dag.py:83: error: Argument 2 to "assertIncludes" of "TestCase" has incompatible type "list[str]"; expected "AbstractSet[Never]" [arg-type] tests/storage/test_msc4242_state_dag.py:90: error: Argument 1 to "assertIncludes" of "TestCase" has incompatible type "list[str]"; expected "AbstractSet[Never]" [arg-type] tests/storage/test_msc4242_state_dag.py:90: error: Argument 2 to "assertIncludes" of "TestCase" has incompatible type "list[Any]"; expected "AbstractSet[Never]" [arg-type] tests/storage/test_msc4242_state_dag.py:103: error: Argument 1 to "assertIncludes" of "TestCase" has incompatible type "list[str]"; expected "AbstractSet[Never]" [arg-type] tests/storage/test_msc4242_state_dag.py:103: error: Argument 2 to "assertIncludes" of "TestCase" has incompatible type "list[Any]"; expected "AbstractSet[Never]" [arg-type] tests/storage/test_msc4242_state_dag.py:107: error: Argument 1 to "assertIncludes" of "TestCase" has incompatible type "list[str]"; expected "AbstractSet[Never]" [arg-type] tests/storage/test_msc4242_state_dag.py:107: error: Argument 2 to "assertIncludes" of "TestCase" has incompatible type "list[Any]"; expected "AbstractSet[Never]" [arg-type]

..because it wants a set not a list. So going back to assertEqual.

Creating a set from a list is as easy as set(...).

want_new_extrems is already a set and new_extrems is already converted to a set. Should work

synapse/tests/storage/test_msc4242_state_dag.py

Lines 162 to 185 in d59f31a

def _test(

self,

current_fwds: list[str],

new_events: list[tuple[FrozenEventVMSC4242, EventContext]],

want_new_extrems: set[str],

want_raises: bool = False,

) -> None:

"""

Tests the logic of _calculate_new_state_dag_extremities.

Tests that the new extremities calculated as a result of processing current_fwds and new_events

matches want_new_extrems or raises if want_raises is True.

"""

coroutine = self.persistence._calculate_new_state_dag_extremities(

self.room_id,

frozenset(current_fwds),

new_events,

)

if want_raises:

f = self.get_failure(coroutine, SynapseError)

assert f is not None

return

new_extrems = set(self.get_success(coroutine))

Resolved without change

tests/storage/test_msc4242_state_dag.py

tests/test_event_auth.py

synapse/storage/controllers/persist_events.py

…pport

Spawning from #19424 (comment)

MadLittleMods · 2026-03-18T15:09:53Z

synapse/event_auth.py

+            known_prev_state_event_ids = set(prev_state_events)
+            raise AssertionError(
+                f"Event {event.event_id} has unknown prev_state_events "
+                + f"{len(prev_state_events)} != {len(prev_state_events_ids)} "


To avoid some of the jumble of output we can group this together better with a description.

Suggested change

+ f"{len(prev_state_events)} != {len(prev_state_events_ids)} "

+ f"({len(prev_state_events)}/{len(prev_state_events_ids)} known)"

MadLittleMods · 2026-03-18T15:16:25Z

synapse/api/room_versions.py

+    # Events are always processed in causal order without any gaps in the DAG
+    # (prev_state_events are always known), guaranteeing that processed events have a path to the
+    # create event. This is an emergent property of state DAGs as asserting that there is a path
+    # to the create event every time we insert an event would be prohibitively expensive.
+    # This is similar to how doubly-linked lists can potentially not refer to previous items correctly
+    # without verifying the list's integrity, but doing it on every insert is too expensive.


I think I would also expect this to live in _calculate_new_state_dag_extremities where we deal with missing prev events.

Previous discussion #19424 (comment)

MadLittleMods · 2026-03-18T15:19:42Z

tests/storage/test_msc4242_state_dag.py

+        second_id = self.helper.send(self.room_id, body="test2")["event_id"]
+        second_prev_state_event = self._get_prev_state_events(second_id)
+        assert len(second_prev_state_event) == 1
+        self.assertEquals(first_prev_state_event, second_prev_state_event)


Using something like self.assertIncludes(..., exact=True) (or self.assertCountEqual(...)) aligns our assertion with what we care about (we don't care about order). People don't have to ask the question about whether we have to preserve the that part of the assertion through refactors.

MadLittleMods · 2026-03-18T15:44:09Z

synapse/handlers/admin.py

+                        # TODO(kegan): better way to typecast and get at this field?
+                        prev_state_events = event.get_dict().get(
+                            "prev_state_events", None
+                        )


TODO

Can we use assert isinstance(event, FrozenEventVMSC4242) here?

MadLittleMods · 2026-03-18T16:00:05Z

synapse/handlers/room.py

+        # the most recently created state event
+        prev_state_event: list[str] | None = (


Suggested change

# the most recently created state event

prev_state_event: list[str] | None = (

# This should be the most recently created state event as we create each event

prev_state_events: list[str] | None = (

MadLittleMods · 2026-03-18T16:04:17Z

synapse/state/__init__.py

        )
        return await ret.get_state(self._state_storage_controller, state_filter)

+    # TODO: Remove as this is unused


MadLittleMods · 2026-03-18T16:06:52Z

synapse/storage/controllers/persist_events.py

-                    ) = await self._calculate_new_forward_extremities_and_state_delta(
-                        room_id, chunk
-                    )
+                if room_version and room_version.msc4242_state_dags:


Redundant

Suggested change

if room_version and room_version.msc4242_state_dags:

if room_version.msc4242_state_dags:

MadLittleMods · 2026-03-18T16:22:25Z

synapse/storage/controllers/persist_events.py

+# The number of forward extremities for each new event.
+msc4242_state_dag_forward_extremities_counter = Histogram(
+    "synapse_storage_msc4242_state_dag_forward_extremities_persisted",
+    "Number of forward extremities for each new event in the state DAG",
+    labelnames=[SERVER_NAME_LABEL],
+    buckets=(1, 2, 3, 5, 7, 10, 15, 20, 50, 100, 200, 500, "+Inf"),
+)


Perhaps we should update contrib/grafana/synapse.json with a matching graph (can copy from the existing graphs that use synapse_storage_events_forward_extremities_persisted)

MadLittleMods · 2026-03-18T16:26:47Z

synapse/storage/schema/main/delta/93/04_state_dag_fwd_extrems.sql

+CREATE INDEX msc4242_state_dag_edges_by_room ON msc4242_state_dag_edges(room_id);
+CREATE UNIQUE INDEX msc4242_state_dag_edges_key ON msc4242_state_dag_edges(room_id, event_id, prev_state_event_id);


I think because the second index has room_id as the first column, it will cover the first index already

MadLittleMods · 2026-03-18T16:27:42Z

synapse/event_auth.py

+        )
+        prev_state_events.update(fetched_prev_state_events)
+        if len(prev_state_events) != len(prev_state_events_ids):
+            # we should have all the prev state events by now, so if we do not, that suggests


Suggested change

# we should have all the prev state events by now, so if we do not, that suggests

# we should have all the `prev_state_events` by now, so if we do not, that suggests

MadLittleMods · 2026-03-18T16:27:59Z

synapse/event_auth.py

+        prev_state_events.update(fetched_prev_state_events)
+        if len(prev_state_events) != len(prev_state_events_ids):
+            # we should have all the prev state events by now, so if we do not, that suggests
+            # a synapse programming error


Suggested change

# a synapse programming error

# a Synapse programming error

MadLittleMods · 2026-03-18T16:30:02Z

tests/federation/test_federation_server.py

        self.assertEqual(channel.json_body["errcode"], "M_FORBIDDEN")

-    @parameterized.expand([(k,) for k in KNOWN_ROOM_VERSIONS.keys()])
+    # Exclude MSC4242 room versions whilst it lacks federation support


Suggested change

# Exclude MSC4242 room versions whilst it lacks federation support

# FIXME: Exclude MSC4242 room versions whilst it lacks federation support

MadLittleMods · 2026-03-18T16:30:09Z

tests/federation/test_federation_server.py

        self._test_get_extremities_common(room_version)

-    @parameterized.expand([(k,) for k in KNOWN_ROOM_VERSIONS.keys()])
+    # Exclude MSC4242 room versions whilst it lacks federation support


Suggested change

# Exclude MSC4242 room versions whilst it lacks federation support

# FIXME: Exclude MSC4242 room versions whilst it lacks federation support

kegsay requested a review from a team as a code owner February 3, 2026 09:33

Changelog

dc3db60

kegsay mentioned this pull request Feb 3, 2026

MSC4242: State DAGs (Federation) #19425

Open

3 tasks

AssertEqual not plural because twisted/twisted#12566

fc4975f

kegsay mentioned this pull request Feb 3, 2026

MSC4242: State DAGs matrix-org/matrix-spec-proposals#4242

Open

MadLittleMods reviewed Feb 25, 2026

View reviewed changes

MadLittleMods reviewed Feb 26, 2026

View reviewed changes

synapse/handlers/message.py Show resolved Hide resolved

MadLittleMods reviewed Feb 26, 2026

View reviewed changes

tests/storage/test_msc4242_state_dag.py Outdated Show resolved Hide resolved

kegsay and others added 11 commits March 9, 2026 14:22

Merge branch 'develop' into kegan/4242-csapi

b69986d

Swap Optional for |: #19424 (comment)

c561c0d

Additional comments per code review

3dff3ac

Apply suggestions from code review

aa46122

Co-authored-by: Eric Eastwood <erice@element.io>

Review comments in persist_events.py

2f82a5b

Additional review comments: non-test files

606ce6b

Review comments in events/builder.py

7311f92

Update synapse/handlers/admin.py

85fa59e

Co-authored-by: Eric Eastwood <erice@element.io>

Update synapse/handlers/message.py

332fd2d

Co-authored-by: Eric Eastwood <erice@element.io>

Update synapse/state/__init__.py

4ea71e0

Co-authored-by: Eric Eastwood <erice@element.io>

More code review (non-test files)

7a1be81

MadLittleMods reviewed Mar 9, 2026

View reviewed changes

synapse/events/__init__.py Outdated Show resolved Hide resolved

MadLittleMods reviewed Mar 9, 2026

View reviewed changes

synapse/handlers/admin.py Outdated Show resolved Hide resolved

MadLittleMods reviewed Mar 9, 2026

View reviewed changes

synapse/storage/controllers/persist_events.py Show resolved Hide resolved

MadLittleMods reviewed Mar 9, 2026

View reviewed changes

synapse/storage/controllers/persist_events.py Show resolved Hide resolved

MadLittleMods reviewed Mar 9, 2026

View reviewed changes

synapse/storage/controllers/persist_events.py Outdated Show resolved Hide resolved

MadLittleMods reviewed Mar 9, 2026

View reviewed changes

synapse/event_auth.py Outdated Show resolved Hide resolved

kegsay and others added 3 commits March 10, 2026 14:49

Review comments: test code and a few extras

d96737b

Update synapse/events/__init__.py

c9d8d9f

Co-authored-by: Eric Eastwood <erice@element.io>

Update synapse/handlers/admin.py

2f8fece

Co-authored-by: Eric Eastwood <erice@element.io>

kegsay and others added 6 commits March 10, 2026 15:03

Update synapse/storage/controllers/persist_events.py

bc91229

Co-authored-by: Eric Eastwood <erice@element.io>

Update synapse/storage/controllers/persist_events.py

6ecf821

Co-authored-by: Eric Eastwood <erice@element.io>

Update synapse/event_auth.py

ecc9860

Co-authored-by: Eric Eastwood <erice@element.io>

Review comments

a0711ef

Use shortstr for consistency with other log messages

85fc75c

Linting

f790d3a

kegsay requested a review from MadLittleMods March 13, 2026 13:02

MadLittleMods added a commit that referenced this pull request Mar 16, 2026

Add MSC3820 context to RoomVersion attributes

b8437a5

Spawning from #19424 (comment)

This was referenced Mar 16, 2026

Add MSC3820 comment context to RoomVersion attributes #19577

Merged

MSC4242: State DAGs matrix-org/complement#841

Open

MadLittleMods reviewed Mar 17, 2026

View reviewed changes

kegsay added 5 commits March 17, 2026 10:21

Review comments

206c77a

Review comments

0b155b7

Merge branch 'develop' into kegan/4242-csapi

cdf9054

Exclude MSC4242 room versions whilst this branch doesn't have fed su…

3a51f74

…pport

Linting

d59f31a

kegsay requested a review from MadLittleMods March 18, 2026 07:58

MadLittleMods added a commit that referenced this pull request Mar 18, 2026

Add MSC3820 comment context to RoomVersion attributes (#19577)

3d960d8

Spawning from #19424 (comment)

MadLittleMods reviewed Mar 18, 2026

View reviewed changes

	def _test(
	self,
	current_fwds: list[str],
	new_events: list[tuple[FrozenEventVMSC4242, EventContext]],
	want_new_extrems: set[str],
	want_raises: bool = False,
	) -> None:
	"""
	Tests the logic of _calculate_new_state_dag_extremities.

	Tests that the new extremities calculated as a result of processing current_fwds and new_events
	matches want_new_extrems or raises if want_raises is True.
	"""
	coroutine = self.persistence._calculate_new_state_dag_extremities(
	self.room_id,
	frozenset(current_fwds),
	new_events,
	)
	if want_raises:
	f = self.get_failure(coroutine, SynapseError)
	assert f is not None
	return

	new_extrems = set(self.get_success(coroutine))

	+ f"{len(prev_state_events)} != {len(prev_state_events_ids)} "
	+ f"({len(prev_state_events)}/{len(prev_state_events_ids)} known)"

		# the most recently created state event
		prev_state_event: list[str] \| None = (

	if room_version and room_version.msc4242_state_dags:
	if room_version.msc4242_state_dags:

		CREATE INDEX msc4242_state_dag_edges_by_room ON msc4242_state_dag_edges(room_id);
		CREATE UNIQUE INDEX msc4242_state_dag_edges_key ON msc4242_state_dag_edges(room_id, event_id, prev_state_event_id);

	# we should have all the prev state events by now, so if we do not, that suggests
	# we should have all the `prev_state_events` by now, so if we do not, that suggests

	# Exclude MSC4242 room versions whilst it lacks federation support
	# FIXME: Exclude MSC4242 room versions whilst it lacks federation support

Conversation

kegsay commented Feb 3, 2026 • edited by MadLittleMods Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kegsay Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MadLittleMods Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

kegsay commented Feb 3, 2026 •

edited by MadLittleMods

Loading

kegsay Mar 17, 2026 •

edited

Loading

MadLittleMods Mar 17, 2026 •

edited

Loading

MadLittleMods Mar 18, 2026 •

edited

Loading