Release 0.21

[BigLep]

Meta

• Release owner: @hacdias
• Release engineer: @hacdias
• Release reviewer: @Jorropo
• Expected RC date: 2023-06-01 2023-06-08 2023-06-12
• 🚢 Expected final release date: 2023-06-08 2023-06-15 2023-07-04
• Changelog: https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.21.md
• Accompanying PR for improving the release process: (example: docs: release process updates from v0.17.0 #9391)
• Accompanying Issue for updating the infra: (example: https://github.com/protocol/bifrost-infra/issues/2221)

Items in scope

Required for RC1

• Accelerated DHT improvements: Move AcceleratedDHTClient from being experimental #9703 - This is an operational improvement item that has been open for multiple months and needs to be closed out for operational hygeine.
• Routing table resilience items: fix: don't add unresponsive DHT servers to the Routing Table libp2p/go-libp2p-kad-dht#820 - This is an operational improvement item that has been open for multiple months and needs to be closed out for operational hygeine.
• Async pinner changes: pin: follow async pinner changes #9859 - Community contribution that has been open for a month plus that publicly said would include.
• go-libp2p update to 0.28 if it's ready (see comment), and if not use 0.27.5
• Boxo release with latest changes Release 0.9.0 boxo#325 - Much of this Kubo release depends on improvements there. In addition, we need to address v0.20.0-rc2: cannot build plugins #9839 since that slipped the last release.
• If secp256k1 key support is missing pem-pkcs8-cleartext import/export #9912 isn't fixed by the time of final release, then revert 67e1a17 and cc original author.
• fix: correct list of addresses for delegated routing #9920 (fix for HTTP PUT provided by IPNI team)
• dag import error handling fixes fix(cmd): useful errors in dag import #9945 moved to 0.22
• dag import not pinning by default + regression tests feat!: dag import - don't pin roots by default #9926 moved to 0.22

Required for RC2

• feat!: dag import - don't pin roots by default #9926
• fix(cmd): useful errors in dag import #9945
• fix: fix abba bug in UsefullNewPeer libp2p/go-libp2p-kbucket#122
• fix: issues discovered in kubo v0.21.0-rc1 libp2p/go-libp2p-kad-dht#851
• feat: webui@4.0.1 #9940
• go-libp2p update to 0.27.6

Required for RC3

• fix(gateway): allow CAR trustless requests with path boxo#358
• go-libp2p update to 0.27.7 chore: bump go-libp2p-kad-dht for deadlock fix #9980

Others

These are items that we had stated at the beginning of the release that we said we wanted to include. We aren't blocking the relse on these. There are also items in 0.21 iteration.

• HTTP Routing v1 streaming support: routing/http: feat: add streaming support boxo#18
• Prepare to deprecate pubsub APIs: Deprecate then Remove /api/v0/pubsub/* RPC API and ipfs pubsub Commands #9717
• Polishing process items of Boxo
• Landing changes for caching proxy for HTTP Routing v1

✅ Release Checklist

Labels

If an item should be executed for a specific release type, it should be labeled with one of the following labels:

• execute ONLY when releasing a Release Candidate
• execute ONLY when releasing a Final Release

Otherwise, it means it should be executed for ALL release types.

Patch releases should follow the same process as .0 releases. If some item should NOT be executed for a Patch Release, it should be labeled with:

• do NOT execute when releasing a Patch Release

Before the release

This section covers tasks to be done ahead of the release.

• Verify you have access to all the services and tools required for the release
  • GPG signature configured in local git and in GitHub
  • admin access to IPFS Discourse
    • ask the previous release owner (or @2color) for an invite
  • access to #shared-pl-marketing-requests channel in FIL Slack
    • ask the previous release owner for an invite
  • access to IPFS network metrics dashboards in Grafana
    • open an access request in the pldw
    • example
  • kuboreleaser checked out on your system (only if you're using kuboreleaser)
  • Thunderdome checked out on your system and configured (see the Thunderdome release docs for setup)
  • docker installed on your system (only if you're using kuboreleaser)
  • npm installed on your system (only if you're NOT using kuboreleaser)
  • zsh installed on your system
  • kubo checked out under $(go env GOPATH)/src/github.com/ipfs/kubo
    • you can also symlink your clone to the expected location by running mkdir -p $(go env GOPATH)/src/github.com/ipfs && ln -s $(pwd) $(go env GOPATH)/src/github.com/ipfs/kubo
  • Reddit account
• Upgrade Go used in CI to the latest patch release available in CircleCI in:
  • ipfs/distributions
    • example
  • ipfs/ipfs-docs
    • example
• Verify there is nothing left for release
• Create a release process improvement PR
  • update the release issue template as you go
  • link it in the Meta section

The release

This section covers tasks to be done during each release (current: 0.21.0)

• Prepare the release branch and update version numbers accordingly
  using kuboreleaser release --version vX.Y.Z(-rcN) prepare-branch or ...

  • create a new branch release-vX.Y.Z
    • use master as base if Z == 0
    • use release as base if Z > 0
  • update the CurrentVersionNumber in version.go in the master branch to vX.Y+1.0-dev
    • example
  • update the CurrentVersionNumber in version.go in the release-vX.Y branch to vX.Y.Z(-RCN)
    • example
  • create a draft PR from release-vX.Y to release
    • example
  • Cherry-pick commits from master to the release-vX.Y.Z using git cherry-pick -x <commit>
  • Add full changelog and contributors to the changelog
    • Replace the Changelog and Contributors sections of the changelog with the stdout of ./bin/mkreleaselog
      • do NOT copy the stderr
  • verify all CI checks on the PR from release-vX.Y to release are passing
  • Merge the PR from release-vX.Y to release using the Create a merge commit
    • do NOT use Squash and merge nor Rebase and merge because we need to be able to sign the merge commit
    • do NOT delete the release-vX.Y branch
• @Jorropo Run Thunderdome testing, see the Thunderdome release docs for details
  • create a PR and merge the experiment config into Thunderdome
• Create the release tag
  using kuboreleaser release --version vX.Y.Z(-rcN) tag or ...

  • This is a dangerous operation! Go and Docker publishing are difficult to reverse! Have the release reviewer verify all the commands marked with ⚠️!
  • ⚠️ tag the HEAD commit using git tag -s vX.Y.Z(-RCN) -m 'Prerelease X.Y.Z(-RCN)'
  • ⚠️ tag the HEAD commit of the release branch using git tag -s vX.Y.Z(-RCN) -m 'Release X.Y.Z(-RCN)'
  • ⚠️ verify the tag is signed and tied to the correct commit using git show vX.Y.Z(-RCN)
  • ⚠️ push the tag to GitHub using git push origin vX.Y.Z(-RCN)
    • do NOT use git push --tags because it pushes all your local tags
• Publish the release to DockerHub
  using kuboreleaser --skip-check-before --skip-run release --version vX.Y.Z(-rcN) publish-to-dockerhub or ...

  • Wait for Publish docker image workflow run initiated by the tag push to finish
  • verify the image is available on Docker Hub
• Publish the release to ipfs.tech
  using kuboreleaser release --version vX.Y.Z(-rcN) publish-to-distributions or ...

  • check out ipfs/distributions
  • run ./dist.sh add-version kubo vX.Y.Z(-RCN) to add the new version to the versions file
    • usage
  • create and merge the PR which updates dists/kubo/versions and dists/go-ipfs/versions ( and dists/kubo/current_version and dists/go-ipfs/current_version)
    • example
  • wait for the CI workflow run initiated by the merge to master to finish
  • verify the release is available on dist.ipfs.io
• Publish the release to NPM
  using kuboreleaser release --version vX.Y.Z(-rcN) publish-to-npm or ...

  • run the Release to npm workflow
  • check Release to npm workflow run logs to verify it discovered the new release
  • verify the release is available on NPM
• Publish the release to GitHub
  using kuboreleaser release --version vX.Y.Z(-rcN) publish-to-github or ...

  • create a new release on GitHub
    • RC example
    • FINAL example
    • use the vX.Y.Z(-RCN) tag
    • link to the release issue
    • link to the changelog in the description
    • check the This is a pre-release checkbox
    • copy the changelog (without the header) in the description
    • do NOT check the This is a pre-release checkbox
  • run the sync-release-assets workflow
  • wait for the sync-release-assets workflow run to finish
  • verify the release assets are present in the GitHub release
• Promote the release
  using kuboreleaser release --version vX.Y.Z(-rcN) promote or ...

  • create an IPFS Discourse topic
    • prerelease example
    • release example
    • use Kubo vX.Y.Z(-RCN) is out! as the title
    • use kubo and go-ipfs as topics
    • repeat the title as a heading (##) in the description
    • link to the GitHub Release, binaries on IPNS, docker pull command and release notes in the description
  • pin the IPFS Discourse topic globally
    • you can make the topic a banner if there is no banner already
  • verify the IPFS Discourse topic was copied to:
    • #ipfs-chatter in IPFS Discord
    • #ipfs-chatter in FIL Slack
    • #ipfs-chatter:ipfs.io in Matrix
  • Add the link to the IPFS Discourse topic to the GitHub Release description
    • example
  • create an issue comment mentioning early testers on the release issue
    • example
  • create an issue comment linking to the release on the release issue
    • example
  • ask the marketing team to tweet about the release in #shared-pl-marketing-requests in FIL Slack
    • example
  • post the link to the GitHub Release to Reddit
    • example
• Test the new version with ipfs-companion
  using kuboreleaser release --version vX.Y.Z(-rcN) test-ipfs-companion or ...

  • run the e2e
    • use vX.Y.Z(-RCN) as the Kubo image version
  • wait for the e2e workflow run to finish
• Update Kubo in interop
  using kuboreleaser release --version vX.Y.Z(-rcN) update-interop or ...

  • check out ipfs/interop
  • run npm install
  • create a PR which updates package.json and package-lock.json
  • merge the PR
• Update Kubo in ipfs-desktop
  using kuboreleaser release --version vX.Y.Z(-rcN) update-ipfs-desktop or ...

  • check out ipfs/ipfs-desktop
  • run npm install
  • create a PR which updates package.json and package-lock.json
  • add @SgtPooki and @whizzzkid as reviewers
• Update Kubo docs
  using kuboreleaser release --version vX.Y.Z(-rcN) update-ipfs-docs or ...

  • run the update-on-new-ipfs-tag.yml workflow
  • merge the PR created by the update-on-new-ipfs-tag.yml workflow run
• Create a blog entry on ipfs.tech
  using kuboreleaser release --version vX.Y.Z(-rcN) update-ipfs-blog --date YYYY-MM-DD or ...

  • create a PR which adds a release note for the new Kubo version
    • example
  • merge the PR
  • verify the blog entry was published
• Merge the release branch back into master, ignoring the changes to version.go (keep the -dev) version,
  using kuboreleaser release --version vX.Y.Z(-rcN) merge-branch or ...

  • create a new branch merge-release-vX.Y.Z from release
  • create and merge a PR from merge-release-vX.Y.Z to master
• Prepare for the next release
  using kuboreleaser release --version vX.Y.Z(-rcN) prepare-next or ...

  • Create the next changelog
  • Link to the new changelog in the CHANGELOG.md file
  • Create the next release issue
• Create a dependency update PR
  • check out ipfs/kubo
  • run go get -u in root directory
  • run go mod tidy in root directory
  • run go mod tidy in docs/examples/kubo-as-a-library directory
  • create a PR which updates go.mod and go.sum
  • add the PR to the next release milestone
• Close the release issue

[BigLep]

The issue description has been updated with the updated set of items that are required for the 2023-06-01 RC.

[BigLep]

Issue description has been updated, but we should update to the latest go-libp2p (which is now go-libp2p 0.27.5, rather than 0.27.4).

[BigLep]

Concerning go-libp2p related releases...

We're not going to depend on the upcoming go-libp2p 0.28 release. go-libp2p v0.28.0 will introduce smart dialling. Smart dialling prioritizes LAN addresses when possible so if you know about a LAN address of someone else, you will first ONLY try those. Unfortunately we announce LAN addresses in the DHT, so those dials habr to fail (or a timer has to expire) before we actually try the public IPs to avoid a performance hit on all connection to nodes that are not on your LAN. As a result, we need libp2p/go-libp2p-kad-dht#839 which removes the private IPs from the public DHT. However if we add libp2p/go-libp2p-kad-dht#839 we are breaking the cloud VPC usecase by removing the ability to use private IPs on the DHT. Arguably relying on private IPs over public DHT was a bug, so we should fix it by giving users control over their private DHT too with #9884.

The Kubo release is already falling behind and I want to minimize the scope creep. We need to stick to the required items above. As a result, I think the actions are:

1. Kubo 0.21 depends on go-libp2p 0.27.5 (not 0.28)
2. Kubo 0.21 depends on a go-libp2p-kad-dht release that has fix: don't add unresponsive DHT servers to the Routing Table libp2p/go-libp2p-kad-dht#820 but not filter local addresses (for WAN) and localhost addresses (for LAN) libp2p/go-libp2p-kad-dht#839

Cc @marten-seemann and @guillaumemichel so they're aware

[BigLep]

Update concerning the go-libp2p release per Slack thread...

Kubo can update to go-libp2p 0.28 if it's ready since it won't incure extra work on the maintainers. This is because for go-libp2p 0.28 smart dialing local addresses and public addresses are dialed in parallel. For non-relay addresses, ranking and delaying dials happens within the address groups, not between address groups. It will not first dial localhost & LAN then public then relay. Instead go-libp2p v0.28 will first dial in parallel the best LAN, the best public v4 and the best public v6. There are docs about this in libp2p/go-libp2p#2336.

If go-libp2p 0.28 is ready when we start the Kubo release, we'll pick it up.