add matrix.org homeserver experimental features policy #3089
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| +++ | ||
| title = "Experimental Features" | ||
| weight = 40 | ||
| +++ | ||
|
|
||
| ## Background | ||
|
|
||
| The matrix.org homeserver is a public Matrix online service that can be used by anyone to interact with the public Matrix federation. | ||
| It is provided by [Element Creations Ltd](https://element.io) on behalf of The Matrix.org Foundation and consists of multiple software components. | ||
|
|
||
| [Synapse](https://github.com/element-hq/synapse)/[Synapse Pro](https://element.io/server-suite/synapse-pro), [Sydent](https://github.com/element-hq/sydent) and [MAS](https://github.com/element-hq/matrix-authentication-service) implement the APIs described by the [Matrix Specification](https://spec.matrix.org). | ||
| Changes to the spec can be proposed by using the [MSC process](https://github.com/matrix-org/matrix-spec-proposals#summary-of-the-process). | ||
| (There are also implementation-specific APIs not in the spec, mostly [admin APIs](https://matrix-org.github.io/synapse/develop/usage/administration/admin_api/index.html), which are *not* subject to this policy.) | ||
|
|
||
| Often there is a need to [evaluate changes before an MSC is accepted](https://spec.matrix.org/proposals/#early-release-of-an-mscidea), to ensure that they will work in practice with a good user experience. We refer to such changes as *experimental features*. | ||
| This page discusses the policy for enabling experimental features on the matrix.org homeserver. | ||
|
|
||
| ## Feature Flags | ||
|
|
||
| One way to evaluate experimental features is via a feature branch, but long-lived branches are difficult to manage, so a preferable solution is often to merge it to the project's development mainline, but leave it **disabled by default**. | ||
|
|
||
| Element's Backend team is normally happy to accept pull requests for experimental features behind a configuration flag, so long as the changes meet the code quality standards and the feature is in active development. | ||
| (Though note that the team may have to remove experimental features at short notice if they are causing problems.) | ||
|
|
||
| Once the feature is available in mainline, it can be enabled on individual servers for testing and evaluation. | ||
|
Comment on lines
+20
to
+25
Member
There was a problem hiding this comment. General comment on this doc, and this bit in particular: it feels like it's going a long way beyond its remit of documenting I think, if you want something to put under https://matrix.org/homeserver/, what you should be creating here is quite a different document to the one that I shared with you and you used as a basis for this. That document was written to answer the question "hi I have an exciting feature, where am I allowed to deploy it", which means that it spends a lot of time talking about the early stages of a feature which shouldn't concern matrix.org at all. Instead, I think you want to come at this from the opposite direction: "here's what is allowed on matrix.org, here's what's definitely not allowed, and here's what we'll want your signature in blood that we can turn it off in 3 months. And here are the reasons we require those things." Maybe you could also throw in some pointers to other ways to try out functionality, but I don't think it should be front and centre. |
||
|
|
||
| Depending on the feature, it may also be possible to enable it for *specific users* on `matrix.org` (e.g., the development team, or other known power-users who are happy to take the risk of broken clients). | ||
|
|
||
| ## Widening the Net | ||
|
|
||
| If we're not getting enough traction on development servers, we may want to consider rolling it out further. | ||
|
|
||
| It's worth remembering at this point that, whatever happens to an experimental feature, further changes *will* be needed. | ||
| Hopefully the feature will be a great success, in which case we'll need to standardise the API and replace the `/unstable` endpoints with stable ones. | ||
| Alternatively, we'll need to iterate on the feature, or, in the worst case, remove it altogether. | ||
|
|
||
| Experience shows that once we've enabled an experimental feature on a server as widely-used as `matrix.org`, clients and users come to rely on the changes (even when care is taken to avoid doing so). | ||
| This then makes it very hard to make further changes to the feature — *including replacing the experimental version with a stable one once development is completed* — due to the risk of breaking clients. | ||
|
|
||
| In general therefore, it's much preferred to do the bulk of testing on non–`matrix.org` servers. | ||
| Both `beta.matrix.org` (for public testing) and internal servers are good candidates for doing so. | ||
| This helps avoid many of the above issues. | ||
| (TODO See [more on beta.matrix](https://handbook.element.io/new-vector/internal/-/wikis/Beta.matrix.org).) | ||
|
Collaborator
Author
There was a problem hiding this comment. Is this something also potentially worth publishing?
Member
There was a problem hiding this comment. it's mostly ops stuff: how to push out changes, etc. It's not really worth publishing, no.
Collaborator
Author
There was a problem hiding this comment. Ah, I was hoping for general info about what the Foundation offers at beta.matrix.org, perhaps to go into a "for devs" page under homeserver or some such. The goal for beta.matrix.org isn't very clearly defined beyond what this document says, though perhaps intentionally, and it would be communicated through the blog when people are supposed to test something.
Member
There was a problem hiding this comment. Currently this homeserver instance is used to allow Element app devs to test a feature that we wouldn't otherwise enable on matrix.org due to risk of clients relying on the feature. We could open that up to be a place where any developer can request an unstable feature be hosted to test against. But that'd be something Element would need to consciously commit to maintaining. If we decide to stick with the current purpose, it should probably be under the |
||
|
|
||
| If, having considered the risks, we still want to roll out the feature further, the next step might be to roll it out to a larger subset of users on `matrix.org`, e.g. as part of a community testing initiative. | ||
|
|
||
| Enabling the feature for *all* users on `matrix.org` is a last resort. | ||
|
Collaborator
Author
There was a problem hiding this comment. Is there transparency that allows me to learn the current status at any time? If not, how can we achieve it? Per the spec, one can query
Member
There was a problem hiding this comment. It's not currently exposed publicly, no. For obvious reasons (the fact it contains secret keys, for a start), the synapse configuration isn't public. You could consider whether there is a way to expose the Honestly I think you need to consider whether the work involved in exposing this is worthwhile: what do you hope to gain from it, and are you sure that it is the most important thing for the backend team to be working on?
Member
There was a problem hiding this comment. May be better to create a
Collaborator
Author
There was a problem hiding this comment.
Why not just include everything in |
||
| When doing this we'll need to make sure we consider up-front things like: | ||
|
|
||
| - how we make sure that clients don't end up relying on it by accident (for example, guarding it by a 'labs' flag which users have to enable explicitly). | ||
| - how we will safely evolve the feature if it turns out more changes are needed to the API. | ||
| - how we will safely remove the `/unstable` prefixes once the feature is stabilised. | ||
|
|
||
| We'll generally want to have a written document addressing these concerns, to make sure that everyone is working on the same understanding. | ||
|
Collaborator
Author
There was a problem hiding this comment. Would it be possible to share these publicly?
Member
There was a problem hiding this comment. Probably. Generally I'd expect them to end up in a github issue somewhere.
Collaborator
Author
There was a problem hiding this comment. Is there anything I can do to help this get done? |
||
|
|
||
| ## Turning It On Everywhere | ||
|
|
||
| The final step in rolling out a feature is to turn it on everywhere — in other words, to enable it by default in Synapse mainline. | ||
|
|
||
| The potential costs of rolling out an experimental feature to the ecosystem as a whole are obviously even higher than turning it on on `matrix.org`. | ||
| In general, once a feature gets to this stage, we're essentially saying it needs to become part of the standardised Matrix Spec. | ||
|
|
||
| In other words: enabling experimental features by default isn't something we would normally do. | ||
| Instead, the feature should complete the MSC process, thus transforming it into a stable feature. | ||
|
|
||
| ## Summary | ||
|
|
||
| Experimental features should first be tested using a server other than `matrix.org` (e.g. a development or internal server, or `beta.matrix.org`). | ||
| If further testing is required then it can be enabled for specific users on `matrix.org` (this may not be feasible for all features). | ||
| Finally, if there is a strong need to test against a large corpus of users then it can be enabled for all `matrix.org` users (for a limited period), subject to certain conditions. | ||
|
|
||
| Normally experimental features will not be enabled by default within Synapse. | ||
| To roll out a feature widely, it should go through the MSC process. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is m.org running Synapse pro these days? The API doesn't report it:
(How) can the public see said hotfixes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Synapse Pro currently consists mostly of worker processes rewritten in rust. As such, yes, m.org is using some components from Synapse Pro: you just won't see them by hitting
/_matrix/federation/v1/versionbecause that's still served by (almost) the FOSS python impl.Good question. @erikjohnston, @anoadragon453: now that room v12 is out and public, is there any reason for matrix.org to still be tracking the private repository?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not currently, though the team is hesitant to switch back to a public branch for a couple reasons:
Needless to say, we don't want to "hoard patches". Each one puts a maintenance burden on the team due to git conflicts each week when we update from upstream. It's better for us to port those patches to FOSS or Synapse Pro, than have them sit on a matrix.org-specific branch. In a really ideal world, matrix.org wouldn't even have its own branch, as neither FOSS nor ESS customers really benefit from it.
The
matrix-org-hotfixes-privbranch currently contains:All stuff that really should either be removed or reviewed and merged upstream. It's not stuff that'd be useful to directly copy to another homeserver. It may be useful as a point of reference, and I have seen people use it for that in the past...
But with heading towards k8s, and aiming to mostly eliminate this branch, it makes sense to keep it private for now (while we work to trim it way down). Which initially pained me to write - but given the above, it makes sense to me.
Happy to share other aspects of how the homeserver is operated; details on how it's configured, things we look at during incidents, policies for putting unstable content on it, etc. etc. And if matrix.org appears to be doing something odd, happy to answer questions regarding the code it's running in Element Town Square.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see also #3003