Fix /sync missing membership in state_after (re-introduce)

[MadLittleMods]

This PR was originally only to enable MSC4222 Complement tests (/sync state_after) but after merging the fix PR, we discovered that while the tests pass locally, fail in CI. To unblock the RC, we decided to revert the fix PR (see #19474 (comment) for more info). To better ensure tests actually pass in CI, we're re-introducing the fix here in the same PR that we enable the tests in.

---

Fix /sync missing membership in state_after.

This applies to any scenario where the first membership has a different sender compared to the state_key and then the second membership has the same sender/state_key. Like someone inviting another person and then them joining. Or someone being kicked and then they leave.

This bug has been present since the MSC4222 implementation was introduced into the codebase (#17888).

---

Fix #19455
Fix https://github.com/element-hq/customer-success/issues/656

I have a feeling, this might also fix these issues (will close and see how people report back):

Fix #18182
Fix #19478

Testing strategy

Complement tests: matrix-org/complement#842

We will need #19460 to merge in order to enable the Complement tests in Synapse but this PR should be merged first so they pass in the first place. I've tested locally that the Complement tests pass with this fix.

Dev notes

MSC4222 has already been merged into the spec and is already part of Matrix v1.16 but we haven't stabilized support in Synapse yet.

---

In the same ballpark:

• Member events for sender of events in sync timeline missing from state_after #19455
• /sync incorrectly calculates state changes for non-gappy syncs with lazy-loading #17050
• Inconsistent state events returned by series of incremental /sync calls #17430
• /sync and /members do not return "current state" #16940
• Declined invites remain in Element Web until cache clear #18182
• Leave event missing from state_after #18793
• Incorrect membership state with leave/rejoin cycles and state_after support #19478

---

Docker builds preferring remote image over the local image we just built, #19460 (comment)

containerd image store (storage driver, driver type)

-> #19475

Todo

• Wait for Fix /sync missing membership in state_after #19463 to merge so the Complement tests all pass
• Wait for Fix Complement CI not running against the code from our PRs (remote images being chosen over local) #19475 to merge

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[MadLittleMods]

❌ Complement tests failing in CI, https://github.com/element-hq/synapse/actions/runs/22104293143/job/63882711534#step:8:9902

✅ TestSync/parallel/Initial_sync_with_lazy-loading_room_members_->_public_room_`state_after`_includes_all_members_from_timeline (320ms)
❌ TestSync (11.05s)
❌ TestSync/parallel (0s)
❌ TestSync/parallel/Initial_sync_with_lazy-loading_room_members_->_private_room_`state_after`_includes_all_members_from_timeline (420ms)
      msc4222_test.go:85: Expected to see membership state ([@user-1-alice:hs1]) from every sender in the timeline ([@user-1-alice:hs1 @user-2-bob:hs1]). Got error: ContainSubset: length mismatch, larger=1 smaller=2. join part of the sync response: {"next_batch":"s15_3_0_1_1_1_1_3_0_1_1","account_data":{"events":[{"type":"m.push_rules","content":{"global":{"underride":[{"conditions":[{"kind":"event_match","key":"type","pattern":"m.call.invite"}],"actions":["notify",{"set_tweak":"sound","value":"ring"},{"set_tweak":"highlight","value":false}],"rule_id":".m.rule.call","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"m.room.encrypted"},{"kind":"room_member_count","is":"2"}],"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight","value":false}],"rule_id":".m.rule.encrypted_room_one_to_one","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"m.room.message"},{"kind":"room_member_count","is":"2"}],"actions
  ":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight","value":false}],"rule_id":".m.rule.room_one_to_one","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"m.room.message"}],"actions":["notify",{"set_tweak":"highlight","value":false}],"rule_id":".m.rule.message","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"m.room.encrypted"}],"actions":["notify",{"set_tweak":"highlight","value":false}],"rule_id":".m.rule.encrypted","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"im.vector.modular.widgets"},{"kind":"event_match","key":"content.type","pattern":"jitsi"},{"kind":"event_match","key":"state_key","pattern":"*"}],"actions":["notify",{"set_tweak":"highlight","value":false}],"rule_id":".im.vector.jitsi","default":true,"enabled":true},{"conditions":[{"kind":"room_member_count","is":"2"},{"kind":"event_match","key":"type","pattern":"org.matrix.msc3381.poll.start"}],"a
  ctions":["notify",{"set_tweak":"sound","value":"default"}],"rule_id":".org.matrix.msc3930.rule.poll_start_one_to_one","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"org.matrix.msc3381.poll.start"}],"actions":["notify"],"rule_id":".org.matrix.msc3930.rule.poll_start","default":true,"enabled":true},{"conditions":[{"kind":"room_member_count","is":"2"},{"kind":"event_match","key":"type","pattern":"org.matrix.msc3381.poll.end"}],"actions":["notify",{"set_tweak":"sound","value":"default"}],"rule_id":".org.matrix.msc3930.rule.poll_end_one_to_one","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"org.matrix.msc3381.poll.end"}],"actions":["notify"],"rule_id":".org.matrix.msc3930.rule.poll_end","default":true,"enabled":true}],"sender":[],"room":[],"postcontent":[{"conditions":[{"kind":"io.element.msc4306.thread_subscription","subscribed":false}],"actions":[],"rule_id":".io.element.msc4306.rule.unsubscribed_thread","default":true,"ena
  bled":true},{"conditions":[{"kind":"io.element.msc4306.thread_subscription","subscribed":true}],"actions":["notify",{"set_tweak":"sound","value":"default"}],"rule_id":".io.element.msc4306.rule.subscribed_thread","default":true,"enabled":true}],"content":[{"actions":["notify",{"set_tweak":"highlight"},{"set_tweak":"sound","value":"default"}],"rule_id":".m.rule.contains_user_name","default":true,"pattern":"user-1-alice","enabled":true}],"override":[{"conditions":[],"actions":[],"rule_id":".m.rule.master","default":true,"enabled":false},{"conditions":[{"kind":"event_match","key":"content.msgtype","pattern":"m.notice"}],"actions":[],"rule_id":".m.rule.suppress_notices","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"m.room.member"},{"kind":"event_match","key":"content.membership","pattern":"invite"},{"kind":"event_match","key":"state_key","pattern":"@user-1-alice:hs1"}],"actions":["notify",{"set_tweak":"highlight","value":false},{"set_tweak":"sound","value":"default"}],
  "rule_id":".m.rule.invite_for_me","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"m.room.member"}],"actions":[],"rule_id":".m.rule.member_event","default":true,"enabled":true},{"conditions":[{"kind":"event_property_contains","key":"content.m\\.mentions.user_ids","value":"@user-1-alice:hs1"}],"actions":["notify",{"set_tweak":"highlight"},{"set_tweak":"sound","value":"default"}],"rule_id":".m.rule.is_user_mention","default":true,"enabled":true},{"conditions":[{"kind":"contains_display_name"}],"actions":["notify",{"set_tweak":"highlight"},{"set_tweak":"sound","value":"default"}],"rule_id":".m.rule.contains_display_name","default":true,"enabled":true},{"conditions":[{"kind":"event_property_is","key":"content.m\\.mentions.room","value":true},{"kind":"sender_notification_permission","key":"room"}],"actions":["notify",{"set_tweak":"highlight"}],"rule_id":".m.rule.is_room_mention","default":true,"enabled":true},{"conditions":[{"kind":"sender_notification_permission","key":"
  room"},{"kind":"event_match","key":"content.body","pattern":"@room"}],"actions":["notify",{"set_tweak":"highlight"}],"rule_id":".m.rule.roomnotif","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"m.room.tombstone"},{"kind":"event_match","key":"state_key","pattern":""}],"actions":["notify",{"set_tweak":"highlight"}],"rule_id":".m.rule.tombstone","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"m.reaction"}],"actions":[],"rule_id":".m.rule.reaction","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","pattern":"m.room.server_acl"},{"kind":"event_match","key":"state_key","pattern":""}],"actions":[],"rule_id":".m.rule.room.server_acl","default":true,"enabled":true},{"conditions":[{"kind":"event_property_is","key":"content.m\\.relates_to.rel_type","value":"m.replace"}],"actions":[],"rule_id":".m.rule.suppress_edits","default":true,"enabled":true},{"conditions":[{"kind":"event_match","key":"type","patt
  ern":"org.matrix.msc3381.poll.response"}],"actions":[],"rule_id":".org.matrix.msc3930.rule.poll_response","default":true,"enabled":true}]}}}]},"presence":{"events":[{"type":"m.presence","sender":"@user-2-bob:hs1","content":{"presence":"online","last_active_ago":73,"currently_active":true}},{"type":"m.presence","sender":"@user-1-alice:hs1","content":{"presence":"online","last_active_ago":10,"currently_active":true}}]},"device_one_time_keys_count":{"signed_curve25519":0},"device_unused_fallback_key_types":[],"rooms":{"join":{"!UtkorWPjPSXGoVOBnp:hs1":{"timeline":{"events":[{"type":"m.room.create","sender":"@user-1-alice:hs1","content":{"room_version":"10","creator":"@user-1-alice:hs1"},"state_key":"","origin_server_ts":1771342882434,"unsigned":{"membership":"leave","age":413},"event_id":"$5SPn6T8GvFpDXnK5hMEFP68QGfJsGJVaQhE6fmLTTEw"},{"type":"m.room.member","sender":"@user-1-alice:hs1","content":{"displayname":"user-1-alice","membership":"join"},"state_key":"@user-1-alice:hs1","origin_server_ts":1771342882471,"
  unsigned":{"membership":"join","age":376},"event_id":"$fiuf9QqtqfjmmUir_eXczRtAEBKWl8fKgPUM_tzPUm8"},{"type":"m.room.power_levels","sender":"@user-1-alice:hs1","content":{"users":{"@user-1-alice:hs1":100},"users_default":0,"events":{"m.room.name":50,"m.room.power_levels":100,"m.room.history_visibility":100,"m.room.canonical_alias":50,"m.room.avatar":50,"m.room.tombstone":100,"m.room.server_acl":100,"m.room.encryption":100},"events_default":0,"state_default":50,"ban":50,"kick":50,"redact":50,"invite":50,"historical":100,"m.call.invite":50},"state_key":"","origin_server_ts":1771342882495,"unsigned":{"membership":"join","age":352},"event_id":"$xXlH63AStzesZHXpeVIXRixMjzoigpRyMIVNgbOxcp0"},{"type":"m.room.join_rules","sender":"@user-1-alice:hs1","content":{"join_rule":"public"},"state_key":"","origin_server_ts":1771342882505,"unsigned":{"membership":"join","age":342},"event_id":"$G-wxtpOZreQ3HiuHibu5-hd5j9Xa9PTepProZosB44E"},{"type":"m.room.history_visibility","sender":"@user-1-alice:hs1","content":{"history_visi
  bility":"shared"},"state_key":"","origin_server_ts":1771342882505,"unsigned":{"membership":"join","age":342},"event_id":"$1CfDvwXvOzFf92ahMzjwrELeJbTyLaJ1YEEZ7RmYwsg"},{"type":"m.room.member","sender":"@user-2-bob:hs1","content":{"displayname":"user-2-bob","membership":"join"},"state_key":"@user-2-bob:hs1","origin_server_ts":1771342882639,"unsigned":{"membership":"join","age":208},"event_id":"$t0fMRKZ-361YIYunEW0f7pS5iv_DYYuvgXflQwMJ2TM"}],"prev_batch":"s15_3_0_1_1_1_1_3_0_1_1","limited":false},"org.matrix.msc4222.state_after":{"events":[{"type":"m.room.create","sender":"@user-1-alice:hs1","content":{"room_version":"10","creator":"@user-1-alice:hs1"},"state_key":"","origin_server_ts":1771342882434,"unsigned":{"age":413},"event_id":"$5SPn6T8GvFpDXnK5hMEFP68QGfJsGJVaQhE6fmLTTEw"},{"type":"m.room.history_visibility","sender":"@user-1-alice:hs1","content":{"history_visibility":"shared"},"state_key":"","origin_server_ts":1771342882505,"unsigned":{"age":342},"event_id":"$1CfDvwXvOzFf92ahMzjwrELeJbTyLaJ1YEEZ7RmYwsg"
  },{"type":"m.room.join_rules","sender":"@user-1-alice:hs1","content":{"join_rule":"public"},"state_key":"","origin_server_ts":1771342882505,"unsigned":{"age":342},"event_id":"$G-wxtpOZreQ3HiuHibu5-hd5j9Xa9PTepProZosB44E"},{"type":"m.room.member","sender":"@user-1-alice:hs1","content":{"displayname":"user-1-alice","membership":"join"},"state_key":"@user-1-alice:hs1","origin_server_ts":1771342882471,"unsigned":{"age":376},"event_id":"$fiuf9QqtqfjmmUir_eXczRtAEBKWl8fKgPUM_tzPUm8"},{"type":"m.room.member","sender":"@user-2-bob:hs1","content":{"displayname":"user-2-bob","membership":"join"},"state_key":"@user-2-bob:hs1","origin_server_ts":1771342882639,"unsigned":{"age":208},"event_id":"$t0fMRKZ-361YIYunEW0f7pS5iv_DYYuvgXflQwMJ2TM"},{"type":"m.room.power_levels","sender":"@user-1-alice:hs1","content":{"users":{"@user-1-alice:hs1":100},"users_default":0,"events":{"m.room.name":50,"m.room.power_levels":100,"m.room.history_visibility":100,"m.room.canonical_alias":50,"m.room.avatar":50,"m.room.tombstone":100,"m.room.s
  erver_acl":100,"m.room.encryption":100},"events_default":0,"state_default":50,"ban":50,"kick":50,"redact":50,"invite":50,"historical":100,"m.call.invite":50},"state_key":"","origin_server_ts":1771342882495,"unsigned":{"age":352},"event_id":"$xXlH63AStzesZHXpeVIXRixMjzoigpRyMIVNgbOxcp0"}]},"account_data":{"events":[]},"ephemeral":{"events":[]},"unread_notifications":{"notification_count":0,"highlight_count":0},"summary":{"m.joined_member_count":2,"m.invited_member_count":0,"m.heroes":["@user-2-bob:hs1"]}},"!MhszFGTDeNBvPVFCHQ:hs1":{"timeline":{"events":[{"type":"m.room.create","sender":"@user-1-alice:hs1","content":{"room_version":"10","creator":"@user-1-alice:hs1"},"state_key":"","origin_server_ts":1771342882547,"unsigned":{"membership":"leave","age":300},"event_id":"$otXzXEq1SyV7aQd-LeyogG_7EePDnuYb7B4J1ODLZiw"},{"type":"m.room.member","sender":"@user-1-alice:hs1","content":{"displayname":"user-1-alice","membership":"join"},"state_key":"@user-1-alice:hs1","origin_server_ts":1771342882596,"unsigned":{"members
  hip":"join","age":251},"event_id":"$Tfg5sXl3nXmKATWIdLYSxT1nSbEyYH47C0dESebsoS4"},{"type":"m.room.power_levels","sender":"@user-1-alice:hs1","content":{"users":{"@user-1-alice:hs1":100},"users_default":0,"events":{"m.room.name":50,"m.room.power_levels":100,"m.room.history_visibility":100,"m.room.canonical_alias":50,"m.room.avatar":50,"m.room.tombstone":100,"m.room.server_acl":100,"m.room.encryption":100},"events_default":0,"state_default":50,"ban":50,"kick":50,"redact":50,"invite":0,"historical":100},"state_key":"","origin_server_ts":1771342882627,"unsigned":{"membership":"join","age":220},"event_id":"$VTBcExJNmP9Nk2_HqALQV5HxNvw8wodUJuAsFP6W4nA"},{"type":"m.room.join_rules","sender":"@user-1-alice:hs1","content":{"join_rule":"invite"},"state_key":"","origin_server_ts":1771342882637,"unsigned":{"membership":"join","age":210},"event_id":"$MRZNoK-tRrIpq7GgJCJQD4IT3TuaFNNHLdgmzPkxddA"},{"type":"m.room.history_visibility","sender":"@user-1-alice:hs1","content":{"history_visibility":"shared"},"state_key":"","origi
  n_server_ts":1771342882638,"unsigned":{"membership":"join","age":209},"event_id":"$4S1TQFcVWyny_OjSgBrm0KcF1bS-2KSvSfHfS0B0leM"},{"type":"m.room.guest_access","sender":"@user-1-alice:hs1","content":{"guest_access":"can_join"},"state_key":"","origin_server_ts":1771342882638,"unsigned":{"membership":"join","age":209},"event_id":"$S23lurUbBhjNTRY24ornelycHDmwCZDzvngZ9ZptktA"},{"type":"m.room.member","sender":"@user-1-alice:hs1","content":{"displayname":"user-2-bob","membership":"invite"},"state_key":"@user-2-bob:hs1","origin_server_ts":1771342882743,"unsigned":{"membership":"join","age":104},"event_id":"$Tu-ASSC-mGiV-mHNTccwuxq_qbl4Vryb7C_ynlW3AkI"},{"type":"m.room.member","sender":"@user-2-bob:hs1","content":{"displayname":"user-2-bob","membership":"join"},"state_key":"@user-2-bob:hs1","origin_server_ts":1771342882796,"unsigned":{"replaces_state":"$Tu-ASSC-mGiV-mHNTccwuxq_qbl4Vryb7C_ynlW3AkI","prev_content":{"displayname":"user-2-bob","membership":"invite"},"prev_sender":"@user-1-alice:hs1","membership":"join",
  "age":51},"event_id":"$OZ4NPL2DqGyWhPdWryW4eb9cEaZ0cqEoWftSfsT4EpE"}],"prev_batch":"s15_3_0_1_1_1_1_3_0_1_1","limited":false},"org.matrix.msc4222.state_after":{"events":[{"type":"m.room.create","sender":"@user-1-alice:hs1","content":{"room_version":"10","creator":"@user-1-alice:hs1"},"state_key":"","origin_server_ts":1771342882547,"unsigned":{"age":300},"event_id":"$otXzXEq1SyV7aQd-LeyogG_7EePDnuYb7B4J1ODLZiw"},{"type":"m.room.guest_access","sender":"@user-1-alice:hs1","content":{"guest_access":"can_join"},"state_key":"","origin_server_ts":1771342882638,"unsigned":{"age":209},"event_id":"$S23lurUbBhjNTRY24ornelycHDmwCZDzvngZ9ZptktA"},{"type":"m.room.history_visibility","sender":"@user-1-alice:hs1","content":{"history_visibility":"shared"},"state_key":"","origin_server_ts":1771342882638,"unsigned":{"age":209},"event_id":"$4S1TQFcVWyny_OjSgBrm0KcF1bS-2KSvSfHfS0B0leM"},{"type":"m.room.join_rules","sender":"@user-1-alice:hs1","content":{"join_rule":"invite"},"state_key":"","origin_server_ts":1771342882637,"unsign
  ed":{"age":210},"event_id":"$MRZNoK-tRrIpq7GgJCJQD4IT3TuaFNNHLdgmzPkxddA"},{"type":"m.room.member","sender":"@user-1-alice:hs1","content":{"displayname":"user-1-alice","membership":"join"},"state_key":"@user-1-alice:hs1","origin_server_ts":1771342882596,"unsigned":{"age":251},"event_id":"$Tfg5sXl3nXmKATWIdLYSxT1nSbEyYH47C0dESebsoS4"},{"type":"m.room.power_levels","sender":"@user-1-alice:hs1","content":{"users":{"@user-1-alice:hs1":100},"users_default":0,"events":{"m.room.name":50,"m.room.power_levels":100,"m.room.history_visibility":100,"m.room.canonical_alias":50,"m.room.avatar":50,"m.room.tombstone":100,"m.room.server_acl":100,"m.room.encryption":100},"events_default":0,"state_default":50,"ban":50,"kick":50,"redact":50,"invite":0,"historical":100},"state_key":"","origin_server_ts":1771342882627,"unsigned":{"age":220},"event_id":"$VTBcExJNmP9Nk2_HqALQV5HxNvw8wodUJuAsFP6W4nA"}]},"account_data":{"events":[]},"ephemeral":{"events":[]},"unread_notifications":{"notification_count":0,"highlight_count":0},"summary"
  :{"m.joined_member_count":2,"m.invited_member_count":0,"m.heroes":["@user-2-bob:hs1"]}}}}}

[MadLittleMods]

For reference, these tests pass just fine locally: COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -p 1 ./tests/msc4222 -run 'TestSync$'

[MadLittleMods]

It appears like our CI isn't running tests against the Synapse source in the PR. For example, I just pushed 702c146 which should prevent the entire app from starting but the test still runs. And I don't see any of the other debug logs I added. 🤔

[MadLittleMods]

As far as I can tell, the Docker builds aren't using each other as expected: matrixdotorg/synapse:latest -> matrixdotorg/synapse-workers:latest -> complement-synapse:latest.

ex. https://github.com/element-hq/synapse/actions/runs/22115069502/job/63921120432?pr=19460

• matrixdotorg/synapse:latest: sha256:6fd52ad2f52f02116fdfaefce34a3c00e1beca015eedb8469b2cc9b6e9ce5de2
• matrixdotorg/synapse-workers:latest: sha256:eb1e7a17cb4e022124dab8eb1ef3c042aa27ad3676eafd856a6ca282dbdf73c2
  • Pulls down matrixdotorg/synapse:latest@sha256:657cfa115c71701d188f227feb9d1c0fcd2213b26fcc1afd6c647ba333582634 ❌
• complement-synapse:latest: sha256:b490b638ffcd68b2ad962988ca93f42ef5e6db275485c3d167daaba745d863b4

From the sha256 ID's, it appears like matrixdotorg/synapse:latest builds an image but then matrixdotorg/synapse-workers:latest pulls down a remote version instead of using the local version we just built.

complement-synapse:latest appears to pull the matrixdotorg/synapse-workers:latest image we just built as expected.

---

For reference, @AndrewFerr has run into this kind of problem locally before, see #18210

[MadLittleMods]

When running docker system info in CI, we can see the same problematic overlayfs storage driver and containerd driver type that @AndrewFerr experienced the same problem with locally in #18210

 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1

Full docker system info output

$ docker system info
Client: Docker Engine - Community
 Version:    29.1.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.40.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 29.1.5
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.14.0-1017-azure
 Operating System: Ubuntu 24.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.62GiB
 Name: runnervmjduv7
 ID: 169aa517-19ec-4d49-bd61-4bb79f5886f6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: githubactions
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

[MadLittleMods]

New containerd image store default in Docker

My guess is that this changed with Docker Engine 29.0.0 given:

containerd image store is now the default for fresh installs. This doesn't apply to daemons configured with userns-remap (see moby#47377).

-- 29.0.0 (2025-11-10), https://docs.docker.com/engine/release-notes/29/#2900

And our ubuntu-latest GitHub runner (Current runner version: '2.331.0') points to using Docker client/server 29.1.5 🎯

This Docker version bump happened on actions/runner-images@416418d (2026-02-10) (28.0.4 -> 29.1.5). Specific PR: actions/runner-images#13633

Arghh!!!!!

Yet another reason to use our own custom GitHub runners.

[MadLittleMods]

Fixed in #19475

[MadLittleMods]

Yes, there is another bug here, #19464

We should be adding to the timeline_state right away before checking (EventTypes.Member, event.sender) not in timeline_state

[MadLittleMods]

Thanks for the review @reivilibre and @anoadragon453 on the original PR 🦙