Skip to content

Release Process #2490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ggwpez merged 50 commits into from
Jan 26, 2024
Merged

Release Process #2490

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
11aa7ac
Update README
ggwpez Nov 24, 2023
5eea0df
Add RELEASE doc
ggwpez Nov 24, 2023
d8b7524
Update README
ggwpez Nov 24, 2023
d622a7c
Add AUDIT.md
ggwpez Nov 24, 2023
1e5b3b0
Typo
ggwpez Nov 24, 2023
becbb79
Fix version format
ggwpez Nov 24, 2023
316579d
Update README.md
ggwpez Nov 27, 2023
11b77b0
Apply suggestions from code review
ggwpez Nov 27, 2023
6cdfd59
Remove leeway
ggwpez Nov 27, 2023
35d87bf
Add sections about Testnets and versioning
ggwpez Nov 27, 2023
5189616
Only publish changed crates
ggwpez Nov 27, 2023
a769906
Explain more
ggwpez Dec 1, 2023
6d61040
Apply suggestions from code review
ggwpez Dec 4, 2023
d9bf9ef
Update docs/RELEASE.md
ggwpez Dec 8, 2023
71d9062
Update docs/RELEASE.md
ggwpez Dec 8, 2023
4d65917
Explain more stuff
ggwpez Dec 8, 2023
1cc2780
Update docs/RELEASE.md
ggwpez Dec 8, 2023
423d0ab
Review fixes
ggwpez Dec 8, 2023
9d7bdfd
Merge remote-tracking branch 'origin/oty-release-doc' into oty-releas…
ggwpez Dec 8, 2023
2bdd380
No GH release for nightly
ggwpez Dec 8, 2023
51322b3
SemVer exempt node internal crates
ggwpez Dec 8, 2023
2cd25d9
Exclude unstable APIs
ggwpez Jan 2, 2024
554b247
Minor fixes
ggwpez Jan 2, 2024
d11e585
Improve clobbering script
ggwpez Jan 2, 2024
26d5a8a
Merge remote-tracking branch 'origin/master' into oty-release-doc
ggwpez Jan 8, 2024
7de5e5e
Update release doc
ggwpez Jan 8, 2024
ad657d9
newlines
ggwpez Jan 10, 2024
7158244
Update docs/RELEASE.md
ggwpez Jan 15, 2024
bfad729
Apply suggestions from code review
ggwpez Jan 15, 2024
1e10077
Update README.md
ggwpez Jan 23, 2024
f8dea0b
Update README.md
ggwpez Jan 23, 2024
3341c6b
Update docs/AUDIT.md
ggwpez Jan 23, 2024
3702afb
Update docs/AUDIT.md
ggwpez Jan 23, 2024
dce630d
Update docs/RELEASE.md
ggwpez Jan 23, 2024
c3014c4
Update docs/RELEASE.md
ggwpez Jan 23, 2024
702118b
Update docs/RELEASE.md
ggwpez Jan 23, 2024
ae7e73c
Apply suggestions from code review
ggwpez Jan 23, 2024
7a4b801
Apply suggestions from code review
ggwpez Jan 23, 2024
6169ee8
Mainline -> Stable
ggwpez Jan 23, 2024
d03a2d3
Use uniform numbers
ggwpez Jan 23, 2024
80b81b1
Spelling
ggwpez Jan 23, 2024
9328f76
Wrap lines at 120
ggwpez Jan 23, 2024
e7032c5
Condense bump hint
ggwpez Jan 23, 2024
818c013
Cleanup
ggwpez Jan 23, 2024
f36af66
Merge branch 'master' into oty-release-doc
ggwpez Jan 23, 2024
d01c07a
Update Westend to nightly every 2w
ggwpez Jan 23, 2024
ef1f428
Add hint to Fellowship releases
ggwpez Jan 23, 2024
d11235f
Line breaks
ggwpez Jan 23, 2024
09e539a
Update README.md
ggwpez Jan 24, 2024
fc28e27
MR -> PR
ggwpez Jan 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,21 @@ Polkadot.

Cumulus is a set of tools for writing Substrate-based Polkadot parachains.

## Releases

> [!NOTE]
> Our release process is still Work-In-Progress and may not yet reflect the aspired outline here.

The Polkadot-SDK has two release channels: `mainline` and `nightly`. Production software is advised to only use `mainline`. `nightly` is meant for tinkerers to try out the latest features. The detailed release process is described in [RELEASE.md](docs/RELEASE.md)

### Mainline

`mainline` releases have a support duration of **three months**. In this period, the release will not have any breaking changes but only receive bug and security fixes on a **two week** cadence.

### Nightly

`nightly` releases are released every night with possibly breaking changes and offer **no SemVer** guarantee between each other. They have pre-release version numbers in the format `major.0.0-nightlyYYMMDD`.

## Upstream Dependencies

Below are the primary upstream dependencies utilized in this project:
Expand Down
17 changes: 17 additions & 0 deletions docs/AUDIT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Audit

Audits are conducted to ensure the absence of severe or exploitable bugs. Merge Requests are generally merged into the `master` branch without audit. The `audited` tag is used to track the latest audited commit of the `master` branch. This means that audits need to happen in order of being merged.
This is an optimistic approach that lets us develop with greater speed, while requiring (possibly) large refactors in the failure case.

Audits can be deferred if the logic is gated by an `experimental` feature or marked as "Not Production Ready" within the first line of doc. Such changes should be queued manually before these warnings are removed.

## General Guidelines for what to Audit

There is no single one-fits-all rule. Generally we should audit important logic that could immediately be used on production networks. If in doubt; ask in chat or in the Merge Request.

## Requesting an Audit

1. Add the PR to the project `Security Audit (PRs) - SRLabs`
1. Set status to Backlog
1. Assign priority, considering the universe of PRs currently in the backlog, or just leave it as TBD
1. Add the component
126 changes: 126 additions & 0 deletions docs/RELEASE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Release

The output of a release are the `polkadot` node, runtimes for the Westend & Rococo networks and new versions of the crates published to `crates.io`.

## Versioning

We are releasing multiple different things from this repository in one release, but
we don't want to use the same version for everything. Thus, in the following we explain
the versioning story for the crates, node and Westend & Rococo. To easily refer to a release, we shall use the node version of it.

### Crate

We try to follow SemVer<sup>3</sup> as best as possible for versioning our crates' public APIs.

👉 The public API of our library crates is defined as all public items that are not inside a `__private` module.

### Node

The versioning of the node is done 99% of the time by only incrementing the `minor` version.
The `major` version is only bumped for special releases and the `patch` can be used for an
out of band release that fixes some critical bug. The node version is not following SemVer.
This means that the version doesn't express if there are any breaking changes in the CLI
interface or similar. The node version is declared in the `NODE_VERSION` variable in
`polkadot/node/primitives/src/lib.rs`.
Copy link
Copy Markdown
Contributor

@liamaharon liamaharon Dec 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Why doesn't the node follow semver?
  2. Why do we put the node version in NODE_VERSION instead of just in the Cargo.toml?

Copy link
Copy Markdown
Member

@bkchr bkchr Dec 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Why doesn't the node follow semver?

What would be the point and what would be included in this? RPC? Cli? Some internal changes?

2. Why do we put the node version in NODE_VERSION instead of just in the Cargo.toml?

#1495 (comment)

Copy link
Copy Markdown
Contributor

@liamaharon liamaharon Dec 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the point and what would be included in this? RPC? Cli? Some internal changes?

from the PoV of node operators, with node versions I would expect

  • Patch: I can blindly update my node. No new features, something like a bug fix or perf improvement
  • Minor: I can blindly update my node. Maybe there are some new CLI args / RPC endpoints, but no breakages
  • Major: I should be careful updating my node. There have been breaking changes, it could be something about the CLI args or RPC format has changed

If you ask node operators, I think they would appreciate semver signalling like this?

#1495 (comment)

This doesn't really explain why 🤔

Copy link
Copy Markdown
Contributor

@kianenigma kianenigma Jan 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see Liam's point, that it would have been good to use SemVer to signal the behavior changes of the node.

In my understanding the reason we don't do that is that we have always targeted Polkadot node 1.0 to be the white paper. Now that it has been delivered, we have two options:

  1. Indefinitely keep the major to 1 to adhere to the previous standard.
  2. Pivot into SemVer. All major versions higher than 1 adhere to Polkadot 1. But, then, if someday there is an official Polkadot 2 white paper and node for it, it would be a bit of a pickle.

At the moment, I am leaning towards option 1. I think most operators are already used to reading changelogs and applying any CLI changes.


### Westend & Rococo

For the these networks, we only increment the `spec_version` and sometimes the `transaction_version`. The spec version is also following
the node version. Its schema is: `M_mmm_ppp` and for example `1_002_000` is the node release `1.2.0`. This versioning has no further meaning, and is only done to map from an on chain `spec_version` easily to the release in this repository.

## Backports

Backports should most of the time not be required. We should only backport [critical bug fixes](#bug-and-security-fix) and then release the fixed crates. There should be no need to backport anything from a release branch.

# Processes

The following processes are necessary to actualize our releases. Each process has a *Cadence* on which it must execute and an *Responsible* that is responsible for autonomously doing so and reporting back any error in the RelEng<sup>1</sup> channel.

## Crate Bumping

Cadence: (possibly) each Merge Request. Responsible: Developer that opened the MR.

Following SemVer isn't easy, but there exists [a guide](https://doc.rust-lang.org/cargo/reference/semver.html) in the Rust documentation that explains the small details on when to bump what. This process should be augmented with CI checks that utilize [`cargo-semver-checks`](https://github.com/obi1kenobi/cargo-semver-checks) and/or [`cargo-public-api`](https://github.com/Enselic/cargo-public-api). They must also pay attention to downstream dependencies that require a version bump, because they export the changed API.

### Steps

1. Developer opens a Merge Request with changed crates against `master`.
2. They bump all changed crates according to SemVer.
Copy link
Copy Markdown
Contributor

@alindima alindima Jan 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about having multiple PRs that modify the same crates. I guess we're supposed to bump them only once within a 3-month period right? This can be a bit cumbersome and prone to human error

Copy link
Copy Markdown
Member Author

@ggwpez ggwpez Jan 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes good observation. There was some discussion on if we maybe rather use PrDoc to indicate SemVer bumps, but then we need additional tooling to translate that into crate bumps and it does not play well with other tools.

We would probably need a CI that prevents double bumps, but i agree that it could be a point of friction.

3. They bump all crates that export any changed types in their *public API*.
4. They also bump all crates that inherit logic changes from relying on one of the bumped crates.
Copy link
Copy Markdown
Contributor

@liamaharon liamaharon Dec 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused by this.

Shouldn't crate bumping only be necessary when merging to the release branch, and master bumps are just automated nightly releases?

Do you mean to refer to the release branch instead of master branch here?

Copy link
Copy Markdown
Member Author

@ggwpez ggwpez Dec 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to describe the normal workflow of merging a standard Merge request.

Currently we never touch the Cargo.toml version. But in the future we would need to increment it according to SemVer. This can be a major, minor or patch bump - depending on the change.

Otherwise when would we bump these versions? We dont really "merge" to the release branch, unless its a backport.

Copy link
Copy Markdown
Contributor

@liamaharon liamaharon Dec 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding was

  • merge to master: don't need to do bumps on merge. releases will be made nightly bumping the semver with a new -nightlyYYYY-MM-DD suffix. I don't see benefit of bumping X.Y.Z between nightly releases, because the code will have a breaking bump (from the -nightly change) soon anyway.
  • merge to release: we need to bump according to X.Y.Z semver, because we need to make new releases for the changed crates.

Copy link
Copy Markdown
Contributor

@liamaharon liamaharon Dec 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how I understood it, where master branch only get nightly releases.

each crates versions like this:

stable
v1.0.0 --------> v1.0.1 ------------------------> v2.0.0 ---------------------- -> ...
                                                    ^
                                             CLOBBER|
master                                              |
v2.0.0-nightly2023-01-01 -> ... v2.0.0-nightly2023-04-01 -> v3.0.0-nightly2023-04-02 -> ...

Copy link
Copy Markdown
Member Author

@ggwpez ggwpez Dec 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to bump according to X.Y.Z semver, because we need to make new releases for the changed crates.

But then how do we keep track of these in the meantime? Yes sure we dont need to bump for the nightly releases since they get their own version number overwritten.
We would also need to bullet proof automate these version bump then. I would also like to do this with prdoc and so on, but @bkchr seems to be in favour of just bumping them directly. Note that we dont need to bump multiple times on master if there was no release so far.

Copy link
Copy Markdown
Member

@gavofyork gavofyork Dec 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we would still bump only major every 3 months?

In my proposal, yes.

In my proposal:

  • Major bumps on release every 3 months when the Clobber happens. SemVer respected.
  • Minor bumps on release every 2 weeks for any crate touched by a backport or any crate depending on such a crate. Backport reviews make an explicit check that they do not make breaking changes. SemVer respected.
  • Patch bumps on release for out-of-band security fixes which have been reviewed as non-breaking. SemVer respected.
  • Nightly just has the next major version with a -nightly suffix. SemVer respected.

Copy link
Copy Markdown
Member Author

@ggwpez ggwpez Jan 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding from the forum post is this was the plan yes: master is never bumped manually, it's only bumped by the CI every nightly release.

The nightly is a pre-release, since it has a dash in the version name. It is completely irrelevant for any SemVer considerations and can break at any time.

For example, if a crate version in stable is 1.2.1, master would be set to 2.0.0-nightlyYYYY-MM-DD.

Yes, but only temporarily in the CI. This bump is never actually committed.

Then every 6th release, we remove the -nightlyYYYY-MM-DD from the versions and that becomes our new stable branch.

As you said below, this could introduce major bump without breaking changes. But maybe it would still be a good tradeoff if the other way introduces too much overhead. I mainly dont like it much because it breaks cargo update.

This way, we don't need to worry at all about breaking changes when we merge to master.

We dont have to "worry" about them, but we still have to mark them as breaking.

In my proposal:
Major bumps on release every 3 months when the Clobber happens. SemVer respected.

Just as note: I interpret this as meaning that unnecessary major bump should be avoided, since SemVer does not require them.

Copy link
Copy Markdown
Member Author

@ggwpez ggwpez Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would really first like to try out the tooling and then see what we should do. I mean the tooling probably already works for most of the things we are doing as breaking changes, like changing a function name or removing a parameter.

Okay then lets try the native bumping without Prdoc for now.

Copy link
Copy Markdown
Contributor

@liamaharon liamaharon Jan 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to update this to clarify manual 'crate bumping' applies to backports into stable not master?

Copy link
Copy Markdown
Member Author

@ggwpez ggwpez Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean? We would bump crates in the MR that merges into master.
Then that version bump could get back ported to stable, but its version should not need any tweaking.


## Mainline Release

Cadence: every two weeks. Responsible: Release Team.

This process aims to release the `release` branch as a *Mainline* release every two weeks. It should eventually be automated.

### Steps

1. Check if process [Clobbering](#clobbering) needs to happen and do so, if that is the case.
2. Check out the latest commit of `release`.
3. Verify all CI checks of that commit.
4. Update the `CHANGELOG.md` version and date. <!-- Here we could aggregate the PrDocs into the CHANGELOG -->
5. Open a Merge Request against `release` for visibility.
6. Check if there were any changes since the last release and abort, if not.
7. Run `cargo semver-checks` and `cargo public-api` again to ensure that there are no SemVer breaks.
8. Internal QA from the release team can happen here.
9. Do a dry-run release to ensure that it *should* work.
10. Merge it into `release`.
11. Verify all CI checks.
12. Comment that a *Mainline* release will happen from the merged commit hash.
13. Release all changed crates to crates.io.
14. Create a release on GitHub.

## Nightly Release

Cadence: every day at 00:00 UTC+1. Responsible: Release Team

This process aims to release the `master` branch as a *Nightly* release. The process can start at 00:00 UTC+1 and should automatically do the following steps.

1. Check out the latest commit of branch `master`.
2. Verify all CI checks of that commit.
3. Compare this commit to the latest `nightly*` tag and abort if there are no changes detected.
4. Set the version of all crates to `major.0.0-nightlyYYMMDD` where `major` is the last released `major` version of that crate plus one.
5. Tag this commit as `nightlyYYMMDD`.
9. Do a dry-run release to ensure that it *should* work.
7. Push this tag (the commit will not belong to any branch).
8. Announce the intent to do a *Nightly* release from that tag in the RelEng chat.
9. Release all crates that had changed since the last nightly release to crates.io.
10. Create a release on GitHub.

## Clobbering

Cadence: every 6th release (~3 months). Responsible: Release Team

This process aims to bring branch `release` in sync with the latest audited commit of `master`. It is not done via a Merge Request but rather by just copying files. It should be automated.

The following script is provided to do the clobbering.

```bash
git fetch
git checkout release
git reset --hard origin/audited
git push --force release
```

## Bug and Security Fix

Cadence: n.a. Responsible: Developer

Describes how developers should merge bug and security fixes.

### Steps

1. Developer opens a Merge Request with a bug or security fix.
2. They have the possibility to mark the MR as such, and does so.
3. Audit happens with priority.
4. It is merged into `master`.
5. It is automatically back-ported to `release`.
6. The fix will be released in the next *Mainline* release. In urgent cases, a release can happen earlier.

# Footnotes

1: `RelEng`: The *RelEng: Polkadot Release Coordination* Matrix channel.
2: `General`: The *General* Matrix channel.
3: `SemVer`: Semantic Versioning v2.0.0 as defined on https://semver.org/.