geth immediately closes websocket connection out of the blue

[gnorbsl]

System information

Geth/v1.8.22-stable-7fa3509e/linux-amd64/go1.11.5
OS & Version: Debian Stretch

Expected behaviour

Web3js connects to geth

Actual behaviour

Web3js attempts to connect but socket gets closed immediately with response ' reason: 'Normal connection closure'

Everything worked absolutely fine a day ago and now its impossible to connect to the node over websocket. As long as the script stayed connected it revceived block events, but as soon as I restarted the script without any changes to geth or web3js the websockets drop immediately.

The only error I've found so far is this:

{"jsonrpc":"2.0","error":{"code":-32600,"message":"unexpected EOF"}}

Steps to reproduce the behaviour

install latest stable geth
install latest web3js

[gnorbsl]

I switched to parity with the same web3js version and it works, so I assume this is connected to geth and/or the rinkeby network

[karalabe]

Do you have a repro script that we could try out?

[gnorbsl]

web3 beta41
Geth/v1.8.22-stable-7fa3509e/linux-amd64/go1.11.5
Or latest unstable dockerimage

Rinkeby network

const Web3 = require('web3');

if (typeof web3 !== 'undefined') {
    web3 = new Web3(web3.currentProvider);
} else {
    const wsProvider = new Web3.providers.WebsocketProvider('ws://localhost:8546');

    wsProvider.on('error', e => console.error('WS Error', e));
    wsProvider.on('end', e => console.error('WS End', e));
    web3 = new Web3(wsProvider);
}

web3.eth.subscribe('newBlockHeaders', (error, blockData) => {
    if(error) console.error(error);
    console.log(blockData);
});

web3.eth.subscribe('pendingTransactions', function (error, transactionHash) {
    if(error) console.error(error);
    console.log(transactionHash);
});

[gnorbsl]

This is not a Rinkeby issue, I had the exact same thing happen on our liveserver on the mainnet. I now have to switch to parity, since its impossible to connect to the node over websocket. This seems like a geth issue.

v1.8.21 has the issue too, 1.8.20 doesn't sync because of wrong chain which I expected.

[ekryski]

Agreed. We upgraded to v1.8.23 and it still disconnects randomly. Personally I think it's still related to #16846. I'm pretty sure Geth gets too much data (ie. a smart contract) that it tries to broadcast over the websocket which crashes the socket connection.

And since Web3 still has not published a solution to this (even though there have been a few working PRs) it completes craps out any listening client and won't reconnect.

We've monkey-patched the shit out of this @bidalihq to keep our stuff reliable but this is making it really hard to stay up to date and run things reliably. We were previously on v1.8.2 (🙀) until I did the upgrade this evening to support the hard fork.

With all the other stuff we've got going on I haven't time to seriously dig into the Geth codebase to debug what's happening. Happy to help solve this if you've got some hunches @karalabe. 😄

[ekryski]

I'll add that we haven't seen it on Ropsten either but I don't think it's network related per se. I think it's related to activity on the network.

[holiman]

Is there any open issues in web3 that are related to this ?

[fjl]

This should be fixed in geth 1.9.1, please verify.

[ekryski]

I'll see if we can reproduce. We plan on upgrading to the latest Web3 soon-ish. Might take us a while though.

We had to do some monkey patching to web3 in order to increase the socket buffer size so that it would stop crashing, so our upgrade might be a bit painful.