Spec lazy-loading room members

[turt2live]

Reviewers: Although this has team review requested, I would appreciate particular review from @ara4n as the proposal author and @KitsuneRal as having gone through the gauntlet. Feedback in addition is more than appreciated.

Disclaimer: Although the implementation is lacking in areas (see below), it is believed by the spec core team that lazy-loading has enough benefits to be added to the spec, as per the MSC guidelines.

---

Implementation references:

• initial cut at a room summary API synapse#3574 - Room summary API impl
• speed up /members and add at= and membership params synapse#3568 - /members additions
• make /context lazyload & filter aware synapse#3567 - /context becoming aware of LL
• Implement the lazy_load_members room state filter parameter synapse#2970 - lazy-loading filters + /sync & /messages behaviour

Notable lacks of implementation:

• Lack of pagination token support on /members - https://github.com/matrix-org/synapse/blob/e26e6b3230f0b55376f0f3bf823dd789ac7064d0/synapse/rest/client/v1/room.py#L394
• Synapse always returns redundant members on filtered endpoints - synapse always includes redundant m.room.member events for most endpoints with lazy-loading enabled synapse#5393
• Synapse does not return left/banned users in m.heroes - Synapse doesn't send left members as heroes synapse#4926
• Synapse does not deduplicate redundant members when paginating /messages - https://github.com/matrix-org/synapse/blob/b54b03f9e1abc1964fe5f00115a165a2b8e10df5/synapse/handlers/pagination.py#L262
• Synapse maintains its redundant membership event cache globally but should be per event stream - synapse tracks lazy_loaded_members_cache per user/device but it should be per stream of events synapse#5421

Spec references:

• Spec lazy_load_members and include_redundant_members #1758 - Original spec
• Fixes Need to review LL stuff that merged to the spec and check it describes how to handle heroes etc #1868
• Fixes /rooms/.../members is missing at and not_membership parameters #1945
• MSC1227 - Lazy loading members
• MSC688 - Room summaries (in a lazy loaded world)

Alterations from the proposals:

• The MSCs are unclear of what scope "redundant" is defined under. Written as "across any of the supported endpoints".
• Clarification that servers and clients should be careful when dealing with historical data. Unsure how Synapse/Riot handle this, if at all.
• Unclear if summary should be required. The implementation only does so when lazy-loading is active, however this seems like a bug. Written as required so that clients can use m.heroes to name the room.

(note: due to these proposals being older Google Doc MSCs, they have had the alterations described here represented in the applicable threaded comments instead of the body of the proposal. This is to hopefully more clearly represent the changes.)

[richvdh]

generally looks good. I'm a bit hazy about some of this stuff though :/

[richvdh]

are these properties not required ?

[turt2live]

no, because if they are omitted then the client should interpret it as "no change".

[turt2live]

Shoving this back into the queue to get a checkmark (hopefully) from someone on the team. There's a few people who have reviewed this, so picking on someone isn't super obvious.

[ara4n]

i'm happy with it from my pov fwiw, but i guess my rewrite needs some review if someone can face it.

[turt2live]

arguably this is fine to merge as I reviewed your edits, but I think a third opinion would be good given the amount of review on here.

[ara4n]

let's just merge it.