raft: decouple raft release from etcd release

[ahrtr]

What would you like to be added?

Currently the raft is released & tagged together with etcd. For example, when we release etcd 3.5.5, we will create tag v3.5.5 for etcd and raft/v3.5.5 for raft at the same time respectively.

raft is an important & stable module, it may have different evolution pace from etcd. I think it would be better to qualify and release raft separately, just like what we did for BoltDB.

We probably need to change the module name from go.etcd.io/etcd/raft/v3 to go.etcd.io/raft. Since it's a new beginning, we can release raft starting from v1, which isn't needed in the module name. We need to add the version into the module name starting from v2, namely go.etcd.io/raft/v2. FYI. Module version numbering

We can even create a separate repository for raft, such as github.com/etcd-io/raft. But it isn't a blocker for this request, either way we will release raft separately.

We need to agree on the release criteria of raft if we follow this approach. For the first step, I think we can keep doing what we already have,

1. The datadriven test;
2. etcd workflows;
3. Kubernetes SIG Scalability test. (Note: @serathius maintains etcd version in Kubernetes)

Since release-3.5 is a stable release, so we will not change anything on 3.5 and raft in 3.5. This change is only for the main branch.

So in summary, the request has three points:

1. release raft separately;
2. rename raft module name from go.etcd.io/etcd/raft/v3 to go.etcd.io/raft.
3. (optional) create a separate repository (e.g github.com/etcd-io/raft) for raft.

@hexfusion @mitake @ptabor @serathius @spzala @tbg @nvanbenschoten @pavelkalinnikov Please share your thoughts. thx.

Why is this needed?

see above.

[serathius]

Moving raft to separate repo was already discussed and got early support in etcd maintainer mailing list. Overall it make sense for me as it will allow raft maintainers to implement their changes without being constrained by etcd.

My main worry is about investment cost. Will investing time and effort into code organization is as important now if we could invest it in other places. I think it's up to @tbg to decide if splitting raft would be a net positive investment.

[tbg]

I think it's up to @tbg to decide if splitting raft would be a net positive investment.

The main goal for now should be making it possible for etcd to release with a version of raft of their choosing, i.e. establishing symmetry between etcd and the other consumers of raft (like CockroachDB). Changes to raft should not have to grind to a halt, for example, in the lead-up to an etcd release but this tends to be prudent if both are to be released from a joint SHA.

For the community having more frequent releases of raft could be nice; at CockroachDB we don't depend on releases; we have a good handle on all changes to raft and pick a SHA directly. I am unsure how many other users are out there that would find more frequent releases useful but they definitely wouldn't hurt.

I don't consider it a goal in and of itself to move raft out into a separate repository. This would have some benefits - for example, CI is often flaky and rarely is it ever due to a change made in a PR to raft - but at the same time the additional coverage we're getting through etcd is nice, especially since it involves end-to-end testing of clusters containing different versions of raft.¹

Is it even practical to release two projects out of the same repo? I don't think the Github releases are able to express this very well. Won't they just tar up the contents of the repo as of a single SHA? Doesn't go.mod use git tags to determine which versions exist? So I wonder if just separating the release cycles in practice forces us into a separate repo. I might be missing something that makes it possible to avoid it.

If we go for a separate repo, I think a standard set of Github actions (unit tests, vet, staticcheck) is sufficient - raft is a very vanilla Go library. We can start with the same maintainer sets; I see no reason to split off from etcd any more than necessary by decoupled releases.

Footnotes

1. But even in the same repo, with releases tagged separately, the etcd tests would be using whatever version of raft they pinned, and thus wouldn't exercise the raft code touched in a PR. Even worse, developers might mistakenly think this would still be the case. We could fix this, but then we'd be ~doubling the test suite size, which doesn't seem useful. ↩

[pav-kv]

FWIW, last time I checked it was possible to release different modules independently within the same repo. The magic is in prefixing the tag with the submodule name/path within the repo. For example, see "Tag based versioning" here, and some suggestions in Go docs.

So, you could have v3.x tags for most of the etcd stuff, and in addition raft/v1.0 for raft. Go module resolution would also work.

[pav-kv]

If we end up "restarting" the versioning for the raft module, I would suggest not rushing into v1 straight away though. Use the opportunity and stay in v0 for a bit, reevaluate the API. Once in v1, there is a stricter semver expectation at the code level (not only at the behaviour; e.g. simple renames of exported fields are technically breaking changes). Systems often ignore this rule, but etcd/raft is more at the "library" end, so code-level semver would be ideal.

[tbg]

FWIW, last time I checked it was possible to release different modules independently within the same repo. The magic is in prefixing the tag with the submodule name/path within the repo. For example, see "Tag based versioning" here, and some suggestions in Go docs.

@pavelkalinnikov right this is Go, good to know, but etcd also relies on Github releases: https://github.com/etcd-io/etcd/releases
I don't think there is a way to say "make a release but only of this directory". So awkwardly, an etcd release will contain raft code that isn't actually used in that release. And more awkwardly, a raft release is really a full etcd tarball as well. I don't see a way around that and (not knowing any better) think that the "Github way" of solving this is to "just" have a separate repo.

[tbg]

If we end up "restarting" the versioning for the raft module, I would suggest not rushing into v1 straight away though. Use the opportunity and stay in v0 for a bit, reevaluate the API.

raft is a mature library that is (presumably) used in lots of places, so we would want to have continuity with the current version, i.e. have a production-ready release right away to make it easy for users to switch to. I think it would raise questions to go to v0, or even to v1. Currently we are at v3 and so shouldn't we continue there? I don't see a better way to make clear that the standalone repo is the "latest and greatest". We can write that into docs 100 times, but versions are supposed to be the ground truth.

I understand the desire to evolve the raft API and I am in favor of doing that, but only in a way that adds ~zero friction to existing users. So I would be hesitant to use the repo switch as a reason to make API changes. Besides, I think most API changes we could make would be better done in the same module version, but by providing a new implementation point.

(not only at the behaviour; e.g. simple renames of exported fields are technically breaking changes)

It's a good idea to point out our policy about breaking changes. I would like us to be able to occasionally rename a field, move a struct, etc, and we should be clear on whether that means minting a new major release or not. I tend to think it would be nice to avoid it (don't see the cost-benefit analysis playing out favorably) but open to a discussion. Pavel, since you seem to a) know a lot about versioning and b) seem to care, maybe you could continue leading that discussion?

[ahrtr]

I think it's up to @tbg to decide if splitting raft would be a net positive investment.

I don't agree this. This is a big structure change, nobody can make any unilateral decision. I think it should be up to all existing etcd maintainers' decision, and of course the feedback from the community. If all etcd maintainers reach an agreement, then we should send a discussion topic in etcd-dev to get feedback as well.

Splitting raft has impact on all raft users, but I think it should be positive impact, because raft can be released & maintained separately. etcd doesn't necessarily always to depend on the latest raft code. Picking a specific commit can also work, but it makes more sense to depend on tags.

cc @hexfusion @mitake @ptabor @spzala could you please share your thoughts as well? thx.

I think it would raise questions to go to v0, or even to v1. Currently we are at v3 and so shouldn't we continue there?

Agreed. It makes sense to me.

I don't think there is a way to say "make a release but only of this directory". So awkwardly, an etcd release will contain raft code that isn't actually used in that release. And more awkwardly, a raft release is really a full etcd tarball as well.

The best approach seems to be the third point I mentioned above: create a separate repository for raft, such as github.com/etcd-io/raft. Sounds good?

[ahrtr]

We can start with the same maintainer sets;

Fair enough to me.

[tbg]

The best approach seems to be the third point I mentioned above: create a separate repository for raft, such as github.com/etcd-io/raft. Sounds good?

That's the only clean way I can think of, given the limitations of a Github-supported release process.

[tbg]

We can start with the same maintainer sets;

Fair enough to me.

Just an observation: I'm noticing that our @etcd-io/maintainers-etcd GH team is anchored on the etcd-io org, so hopefully this wouldn't even require maintaining parallel worlds.

[mitake]

I think having a separated raft module is a good idea. Various users of the module can have their own suitable way for controlling risk and benefit from new improvements.

So awkwardly, an etcd release will contain raft code that isn't actually used in that release. And more awkwardly, a raft release is really a full etcd tarball as well.

The best approach seems to be the third point I mentioned above: create a separate repository for raft, such as github.com/etcd-io/raft. Sounds good?

I agree with @ahrtr . Having raft code that isn't actually used in that release seems to be hard to manage. Separating repositories seems to be good.

[spzala]

+1 Any of the three approaches seems good but I think for the long term benefitsgithub.com/etcd-io/raft seems a better approach. We should however make sure that we are good with etcd test stability before we start working on it. Thanks @ahrtr for initiating this discussion.