Release v0.4.23

[Stebalien]

go-ipfs 0.4.23 Release

PR: #6836
Download: https://dist.ipfs.io/go-ipfs/v0.4.23-rc2 (RC2)

Unfortunately, go-ipfs 0.5.0 is taking longer than expected. Given the large number of fixes merged since 0.4.22, we've decided to cut another patch release.

This release contains critical fixes. Please upgrade ASAP. Importantly, we're strongly considering switching to TLS by default in go-ipfs 0.5.0 and dropping SECIO support. However, our the current TLS transport in go-ipfs 0.4.22 has a bug that can cause connections to spontaneously disconnect during the handshake.

This release fixes that bug, among many other issues. Users that don't upgrade may experience connectivity issues when the network upgrades to go-ipfs 0.5.0.

🗺 What's left for release

• Get the tests passing.
• Release Notes.

🚢 Estimated shipping date

January 27th, at the latest (likely earlier).

🔦 Highlights

• Fixes build on go 1.13
• Fixes an issue where we may not connect to providers in bitswap.
• Fixes an issue on our TLS transport where we may abort a handshake unintentionally.
• Fixes a common panic in the websocket transport.
• Adds support for recursively resolving dnsaddrs (makes go-ipfs compatible with the new bootstrappers).
• Fixes several potential panics/crashes.
• Switches to using pre-defined autorelays instead of trying to find them in the DHT:
  • Avoids selecting random, potentially poor, relays.
  • Avoids spamming the DHT with requests trying to find relays.
  • Reduces the impact of accidentally enabling AutoRelay + RelayHop. I.e., the network won't try to DoS you.
• Modifies the connection manager to not count connections in the grace period towards the connection limit.
  • Pro: New connections don't cause us to close useful, existing connections.
  • Con: Libp2p will keep more connections. Consider reducing your HighWater after applying this patch.
• Improved peer usefulness tracking in bitswap. Frequently used peers will be marked as "important" and the connection manager will avoid closing connections to these peers.
• Includes a new version of the WebUI to fix some issues with the peers map.

Changelog

• github.com/ipfs/go-ipfs:
  • feat: update the webui to fix some performance issues (ipfs/go-ipfs#6844)
  • fix: limit SW registration to content root (ipfs/go-ipfs#6801)
  • fix issue 6760, adding with hash-only, high CPU usage. (ipfs/go-ipfs#6764)
  • fix(coreapi/add): close the fake repo used when adding with hash-only (ipfs/go-ipfs#6747)
  • fix bug 6748 (ipfs/go-ipfs#6754)
  • fix(pin): wait till after fetching to remove direct pin (ipfs/go-ipfs#6708)
  • pin: fix pin update X Y where X==Y (ipfs/go-ipfs#6669)
  • namesys: set the correct cache TTL on publish (ipfs/go-ipfs#6667)
  • build: fix golangci again (ipfs/go-ipfs#6641)
  • make: move all test deps to a separate module (ipfs/go-ipfs#6637)
  • fix: close peerstore on stop (ipfs/go-ipfs#6629)
  • build: fix build when we don't have a full git tree (ipfs/go-ipfs#6626)
• github.com/ipfs/go-bitswap (v0.0.8-cbb485998356 -> v0.0.8-e37498cf10d6):
  • fix: wait until we finish connecting before we cancel the context (ipfs/go-bitswap#226)
  • engine: tag peers based on usefulness (ipfs/go-bitswap#191)
• github.com/ipfs/go-cid (v0.0.2 -> v0.0.4):
  • fix parsing issues and nits (ipfs/go-cid#97)
  • Verify that prefix is correct v0 prefix (ipfs/go-cid#96)
• github.com/multiformats/go-multihash (v0.0.5 -> v0.0.10):
  • Ensure that length of multihash is properly handled (multiformats/go-multihash#119)
  • fix murmur3 name (multiformats/go-multihash#115)
  • rename ID to IDENTITY (multiformats/go-multihash#113)
    (multiformats/go-multihash#119)
• github.com/libp2p/go-flow-metrics (v0.0.1 -> v0.0.3):
  • fix bug in meter traversal logic (libp2p/go-flow-metrics#11)
• github.com/libp2p/go-libp2p (v0.0.28 -> v0.0.32):
  • options to configure known relays for autorelay (libp2p/go-libp2p#705)
  • feat(host): recursively resolve addresses (libp2p/go-libp2p#764)
  • mdns: always use interface addresses (libp2p/go-libp2p#667)
• github.com/libp2p/go-libp2p-connmgr (v0.0.6 -> v0.2.1):
  • don't count connections in the grace period against the limit (libp2p/go-libp2p-connmgr#50)
• github.com/libp2p/go-libp2p-kad-dht (v0.0.13 -> v0.0.15):
  • metrics: fix memory leak (libp2p/go-libp2p-kad-dht#390)
• github.com/libp2p/go-libp2p-tls (v0.0.1 -> v0.0.2):
  • close the underlying connection when the handshake fails (libp2p/go-libp2p-tls#39)
  • make the error check for not receiving a public key more explicit (libp2p/go-libp2p-tls#34)
  • Fix: Connection Closed after handshake (libp2p/go-libp2p-tls#37)
• github.com/libp2p/go-libp2p-swarm (v0.0.6 -> v0.0.7):
  • fix: don't assume that transports implement stringer (libp2p/go-libp2p-swarm#134)
• github.com/libp2p/go-ws-transport (v0.0.4 -> v0.0.6):
  • Add mutex for write/close (libp2p/go-ws-transport#65)

Other:

Update bloom filter libraries to remove unsound usage of the unsafe package.

✅ Release Checklist

For each RC published in each stage:

• version string in version.go has been updated
• tag commit with vX.Y.Z-rcN
• upload to dist.ipfs.io
  1. Build: https://github.com/ipfs/distributions#usage.
  2. Pin the resulting release.
  3. Make a PR against ipfs/distributions with the updated versions, including the new hash in the PR comment.
  4. Ask the infra team to update the DNSLink record for dist.ipfs.io to point to the new distribution.
• cut a pre-release on github and upload the result of the ipfs/distributions build in the previous step.

Checklist:

• Stage 0 - Automated Testing
  • Feature freeze. If any "non-trivial" changes (see the footnotes of docs/releases.md for a definition) get added to the release, uncheck all the checkboxes and return to this stage.
  • Automated Testing (already tested in CI) - Ensure that all tests are passing, this includes:
    • unit, sharness, cross-build, etc (make test)
    • lint (make test_go_lint)
    • interop
    • go-ipfs-api
    • go-ipfs-http-client
    • WebUI
• Stage 1 - Internal Testing
  • CHANGELOG.md has been updated
    • use ./bin/mkreleaselog to generate a nice starter list
  • Infrastructure Testing:
    • Deploy new version to a subset of Bootstrappers
      • Deploying to mars.
    • Deploy new version to a subset of Gateways
    • Deploy new version to a subset of Preload nodes
    • Collect metrics every day. Work with the Infrastructure team to learn of any hiccup
  • IPFS Application Testing - Run the tests of the following applications:
    • IPFS Desktop - @hacdias
    • IPFS Companion - @lidel
    • NPM on IPFS
• Stage 2 - Community Dev Testing
  • skipping because this is a patch release
• Stage 3 - Community Prod Testing
  • Documentation
    • Ensure that CHANGELOG.md is up to date
    • Ensure that README.md is up to date
    • Ensure that all the examples we have produced for go-ipfs run without problems
    • Update HTTP-API Documentation on the Website using https://github.com/ipfs/http-api-docs
  • Invite the IPFS early testers to deploy the release to part of their production infrastructure.
  • Invite the wider community through (link to the release issue):
    • discuss.ipfs.io
    • Twitter
    • IRC
• Stage 4 - Release
  • Final preparation
    • Verify that version string in repo/version.go has been updated
    • tag commit with vX.Y.Z
    • update release branch to point to release commit (git merge vX.Y.Z).
    • Release published
      • to dist.ipfs.io
      • to npm-go-ipfs
      • to chocolatey
      • to github
  • Publish a Release Blog post (at minimum, a c&p of this release issue with all the highlights, API changes, link to changelog and thank yous)
  • Broadcasting (link to blog post)
    • Twitter
    • IRC
    • Reddit
    • discuss.ipfs.io
    • Announce it on the IPFS Users Mailing List
• Post-Release
  • Bump the version in version.go to vX.(Y+1).0-dev.
  • Create an issue using this release issue template for the next release.
  • Make sure any last-minute changelog updates from the blog post make it back into the CHANGELOG.

❤️ Contributors

Contributor	Commits	Lines ±	Files Changed
Steven Allen	52	+1866/-578	102
vyzo	12	+167/-90	22
whyrusleeping	5	+136/-52	7
Roman Proskuryakov	7	+94/-7	10
Jakub Sztandera	3	+58/-13	7
hucg	2	+31/-11	2
Raúl Kripalani	2	+7/-33	6
Marten Seemann	3	+27/-10	5
Marcin Rataj	2	+26/-0	5
b5	1	+2/-22	1
Hector Sanjuan	1	+11/-0	1
Yusef Napora	1	+4/-0	1

Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started:

• Check the issues with the help wanted label in the go-ipfs repo
• Join an IPFS All Hands, introduce yourself and let us know where you would like to contribute - https://github.com/ipfs/team-mgmt/#weekly-ipfs-all-hands
• Hack with IPFS and show us what you made! The All Hands call is also the perfect venue for demos, join in and show us what you built
• Join the discussion at discuss.ipfs.io and help users finding their answers.
• Join the 🚀 IPFS Core Implementations Weekly Sync 🛰 and be part of the action!

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode, which is also accessible through our Matrix bridge.

[Stebalien]

@alanshaw could I borrow your expertise at writing wonderful, engaging release notes for this release?

(I understand that might be a bit tricky given that you weren't directly involved in most of the changes; we can discuss this on Monday and/or I can try writing a soulless skeleton).

[Stebalien]

RC1 at /ipfs/QmRxq967po6BcGW68wbUvwin758dkRiejAKmyHDm9wqaRB/go-ipfs/v0.4.23-rc1/. I'll update dist.ipfs.io later.

[alanshaw]

@alanshaw could I borrow your expertise at writing wonderful, engaging release notes for this release?

I'll give it a go!

[lidel]

@Stebalien is it too late to include WebUI fixes from #6778 (comment)?
Commit to cherrypick is 5ab7a70

[Stebalien]

No, we can include that. Could you send a patch against the current RC branch to use the new version?

[Stebalien]

I'm releasing an RC2 with two changes:

1. The backported WebUI update. This should probably count as a "feature" but it's not likely to cause wide-scale regressions and it includes a bunch of important bug fixes.
2. A patch to fix a memory leak from the DHT that I intended to include but didn't. Luckily, we have our handy release checklist and this was caught when profiling a bootstrapper.

[Stebalien]

Early testers: We're entering stage 3 of this patch release. As with the previous release, this release only includes cherry-picked bug fixes (quite a few of them) so we don't expect any regressions or significant performance changes. However, we're still inviting you to do a quick sanity check before the final release.

Note: We have cherry-picked a few patches that could cause interesting behavioral changes:

If you're relying on AutoRelay, we're no longer picking random relays from the DHT. Instead, we've hard-coded a list of known-good relays. This is a stop-gap to deal with the fact that many of the public relays were unreliable, causing nodes using AutoRelay to continuously crawl the network looking for a good relay.

The connection manager no longer counts connections in the "grace period" (default is 30s) against the limit. Before this change, nodes with many inbound connections would disconnect from all long-term peers as connections in the grace period would take the entire quota. You may need to adjust your connection limits to account for this change.

Finally, bitswap now tracks which peers we frequently exchange data with and asks the connection manager to avoid closing connections to these peers.

• Infura (@MichaelMure)
• Textile (@sanderpick)
• Pinata (@obo20)
• RTrade (@postables)
• QRI (@b5)
• Siderus (@koalalorenzo)

[MichaelMure]

Thanks for the release :)

Just a note: consider pinging the early testers when you make the release (aka 5 days ago) instead of at stage 3 so we can do the testing with a bit more room.

[Stebalien]

@MichaelMure you're right, I should have pinged the early testers when creating the release issue. I'll add that to the release process.

[MichaelMure]

What about fixing #6795 as well ?