Nodes Are Unable To Open Outbound Connections Via VPNs

[nisdas]

We have recently updated Prysm to v0.22 from go-libp2p v0.20.3 . However over the course of the release a number of Prysm users running via VPNs mentioned that they were unable to find and connect to any peers. This was only fixed by them falling back to an older prysm release so that they could continue connecting to peers.

After confirming the issue with multiple users, I was able to reproduce this with my own local node when running via a VPN. After looking at all the debug libp2p logs, it became clear that none of the outbound dials were able to reach their remote peer. They all failed this way:

failed to dial 16Uiuabcd:  * [/ip4/xx.xx.xx.xx/tcp/13000] dial tcp4 yy.yy.yy.yy:13000->xx.xx.xx.xx:13000: i/o timeout

I tried updating Prysm to later libp2p releases v0.23.x, however the problem still persisted and Prysm was unable to make any outbound connections if the host machine was running via a VPN. Looking through commits to v0.22 , I found a PR that appeared that it might have touched the dialer #1673 . I reverted the PR on top of v0.22 here
nisdas@3807d6d

and ran prysm with this forked version of libp2p. Prysm was immediately able to find peers and connect with them succesfully. I haven't had time to dig into why exactly it broke for nodes running via VPNs but it does appear that the PR broke it for us.

[marten-seemann]

@nisdas Thank you for digging into this!

@Stebalien (as the author of #1673), any idea what's going on here?

[Stebalien]

• Is the user explicitly listening on specific interfaces instead of 0.0.0.0?
• Is yy.yy.yy.yy the VPN's local IP address?
• I assume the user is listening on 13000?
• Is the user listening on any other ports?

[nisdas]

Is the user explicitly listening on specific interfaces instead of 0.0.0.0?

Yes, most users will not listen in on 0.0.0.0 unless they specifically specify it via a flag. They will bind to a specific network interface instead.

Is yy.yy.yy.yy the VPN's local IP address?

Yes, it is the local ip for the node. The interface that they are binded to.

I assume the user is listening on 13000?

This is correct

Is the user listening on any other ports?

They are not, 13000 is the only port that they listen to.

[MarcoPolo]

@nisdas Do you have a repro steps I could try to follow?