ECONNRESET with HTTP/2 upgrade RST

[chris-laplante]

You want to:

• report a bug
• request a feature

Current behaviour

Application crashes with a ECONNRESET when an HTTP/2 upgrade is attempted (and uncleanly aborted) to a socket on which Socket.IO is listening. Output:

root@host:~# node /usr/lib/node_modules/mymodule/server.py 
server listening on port 3000
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Emitted 'error' event at:
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)

Steps to reproduce (if the current behaviour is a bug)

1. Start this server: https://github.com/chris-laplante/socket.io-fiddle/blob/master/server.js
2. Execute this Python script on a different machine (updating the SERVER_IP variable appropriately):

import socket
import struct

SERVER_IP = "1.2.3.4"

def crash():
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
    s.connect((SERVER_IP, 3000))

    l_onoff = 1
    l_linger = 0
    s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", l_onoff, l_linger))

    # Sending the upgrade header will cause the server to crash
    message = b"HEAD / HTTP/1.1\r\nconnection: Upgrade, HTTP2-Settings\r\nupgrade: h2c\r\nhttp2-settings: AAMAAABkAAQAAP__\r\nHost: www.example.com\r\n\r\n"

    # Sending a normal HEAD without upgrade header won't crash
    #message = b"HEAD / HTTP/1.1\r\nHost: www.example.com\r\n\r\n"

    s.send(message)
    print(s.recv(1000).decode())
    s.close()

if __name__ == "__main__":
    crash()

Expected behaviour

One of the "error" handlers fires. At the very least, the server doesn't crash.

Setup

• OS: Linux 4.9.0
• browser: N/A
• socket.io version: 2.3.0
• Node: v10.16.3

Other information (e.g. stacktraces, related issues, suggestions how to fix)

Output when running the server with NODE_DEBUG="net,http" is below:

root@host:~# NODE_DEBUG="net,http" node /usr/lib/node_modules/mymodule/server.py 
NET 9054: setupListenHandle null 3000 4 0 undefined
NET 9054: setupListenHandle: create a handle
NET 9054: bind to ::
server listening on port 3000
NET 9054: onconnection
NET 9054: _read
NET 9054: Socket._read readStart
HTTP 9054: SERVER new http connection
HTTP 9054: SERVER socketOnParserExecute 127
HTTP 9054: SERVER upgrade or connect HEAD
HTTP 9054: SERVER have listener for upgrade
NET 9054: _final: not ended, call shutdown()
NET 9054: afterShutdown destroyed=false ReadableState {
  objectMode: false,
  highWaterMark: 16384,
  buffer: BufferList { head: null, tail: null, length: 0 },
  length: 0,
  pipes: null,
  pipesCount: 0,
  flowing: null,
  ended: false,
  endEmitted: false,
  reading: true,
  sync: false,
  needReadable: true,
  emittedReadable: false,
  readableListening: false,
  resumeScheduled: false,
  paused: false,
  emitClose: false,
  autoDestroy: false,
  destroyed: false,
  defaultEncoding: 'utf8',
  awaitDrain: 0,
  readingMore: false,
  decoder: null,
  encoding: null }
NET 9054: destroy
NET 9054: close
NET 9054: close handle
NET 9054: has server
NET 9054: SERVER _emitCloseIfDrained
NET 9054: SERVER handle? true   connections? 0
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Emitted 'error' event at:
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)

I have also tried wrapping the application in a domain and catching all 'error' events, then printing them out. The output looks something like this:

NET 13095: onconnection
NET 13095: _read
NET 13095: Socket._read readStart
HTTP 13095: SERVER new http connection
HTTP 13095: SERVER socketOnParserExecute 126
HTTP 13095: SERVER upgrade or connect HEAD
HTTP 13095: SERVER have listener for upgrade
NET 13095: _final: not ended, call shutdown()
NET 13095: afterShutdown destroyed=false ReadableState {
  objectMode: false,
  highWaterMark: 16384,
  buffer: BufferList { head: null, tail: null, length: 0 },
  length: 0,
  pipes: null,
  pipesCount: 0,
  flowing: null,
  ended: false,
  endEmitted: false,
  reading: true,
  sync: false,
  needReadable: true,
  emittedReadable: false,
  readableListening: false,
  resumeScheduled: false,
  paused: false,
  emitClose: false,
  autoDestroy: false,
  destroyed: false,
  defaultEncoding: 'utf8',
  awaitDrain: 0,
  readingMore: false,
  decoder: null,
  encoding: null }
NET 13095: destroy
NET 13095: close
NET 13095: close handle
NET 13095: has server
NET 13095: SERVER _emitCloseIfDrained
NET 13095: SERVER handle? true   connections? 0
{ Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
---------------------------------------------
    at Object.<anonymous> (/usr/lib/node_modules/mymodule/server2.py:15:4)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read',
  domainEmitter:
   Socket {
     connecting: false,
     _hadError: false,
     _handle: null,
     _parent: null,
     _host: null,
     _readableState:
      ReadableState {
        objectMode: false,
        highWaterMark: 16384,
        buffer: BufferList { head: null, tail: null, length: 0 },
        length: 0,
        pipes: null,
        pipesCount: 0,
        flowing: null,
        ended: false,
        endEmitted: false,
        reading: true,
        sync: false,
        needReadable: true,
        emittedReadable: false,
        readableListening: false,
        resumeScheduled: false,
        paused: false,
        emitClose: false,
        autoDestroy: false,
        destroyed: true,
        defaultEncoding: 'utf8',
        awaitDrain: 0,
        readingMore: false,
        decoder: null,
        encoding: null },
     readable: false,
     domain:
      Domain {
        domain: null,
        _events: [Object],
        _eventsCount: 3,
        _maxListeners: undefined,
        members: [],
        [Symbol(kWeak)]: WeakReference {} },
     _events:
      [Object: null prototype] { end: [Function], timeout: [Function] },
     _eventsCount: 2,
     _maxListeners: undefined,
     _writableState:
      WritableState {
        objectMode: false,
        highWaterMark: 16384,
        finalCalled: true,
        needDrain: false,
        ending: true,
        ended: true,
        finished: true,
        destroyed: true,
        decodeStrings: false,
        defaultEncoding: 'utf8',
        length: 0,
        writing: false,
        corked: 0,
        sync: true,
        bufferProcessing: false,
        onwrite: [Function: bound onwrite],
        writecb: null,
        writelen: 0,
        bufferedRequest: null,
        lastBufferedRequest: null,
        pendingcb: 0,
        prefinished: true,
        errorEmitted: true,
        emitClose: false,
        autoDestroy: false,
        bufferedRequestCount: 0,
        corkedRequestsFree: [Object] },
     writable: false,
     allowHalfOpen: true,
     _sockname: null,
     _pendingData: null,
     _pendingEncoding: '',
     server:
      Server {
        domain: [Domain],
        _events: [Object],
        _eventsCount: 5,
        _maxListeners: undefined,
        _connections: 0,
        _handle: [TCP],
        _usingWorkers: false,
        _workers: [],
        _unref: false,
        allowHalfOpen: true,
        pauseOnConnect: false,
        httpAllowHalfOpen: false,
        timeout: 120000,
        keepAliveTimeout: 5000,
        maxHeadersCount: null,
        headersTimeout: 40000,
        _connectionKey: '6::::3000',
        [Symbol(IncomingMessage)]: [Function],
        [Symbol(ServerResponse)]: [Function],
        [Symbol(asyncId)]: 8 },
     _server:
      Server {
        domain: [Domain],
        _events: [Object],
        _eventsCount: 5,
        _maxListeners: undefined,
        _connections: 0,
        _handle: [TCP],
        _usingWorkers: false,
        _workers: [],
        _unref: false,
        allowHalfOpen: true,
        pauseOnConnect: false,
        httpAllowHalfOpen: false,
        timeout: 120000,
        keepAliveTimeout: 5000,
        maxHeadersCount: null,
        headersTimeout: 40000,
        _connectionKey: '6::::3000',
        [Symbol(IncomingMessage)]: [Function],
        [Symbol(ServerResponse)]: [Function],
        [Symbol(asyncId)]: 8 },
     timeout: 120000,
     parser: null,
     on: [Function: socketOnWrap],
     _paused: false,
     [Symbol(asyncId)]: 78,
     [Symbol(lastWriteQueueSize)]: 0,
     [Symbol(timeout)]:
      Timeout {
        _called: false,
        _idleTimeout: -1,
        _idlePrev: null,
        _idleNext: null,
        _idleStart: 167139,
        _onTimeout: null,
        _timerArgs: undefined,
        _repeat: null,
        _destroyed: true,
        domain: [Domain],
        [Symbol(unrefed)]: true,
        [Symbol(asyncId)]: 83,
        [Symbol(triggerId)]: 78 },
     [Symbol(kBytesRead)]: 126,
     [Symbol(kBytesWritten)]: 0 },
  domain:
   Domain {
     domain: null,
     _events:
      [Object: null prototype] {
        removeListener: [Function],
        newListener: [Function],
        error: [Function] },
     _eventsCount: 3,
     _maxListeners: undefined,
     members: [],
     [Symbol(kWeak)]: WeakReference {} },
  domainThrown: false }

[chris-laplante]

Since this repo seems dead, here's the workaround we ended up with:

const { IncomingMessage } = require('_http_incoming');

class HTTP2FilteredIncomingMessage extends IncomingMessage {
    _addHeaderLines(headers, n) {
        const upgradeIndex = headers.findIndex(element => element.trim().toLowerCase() === "upgrade");
        if (upgradeIndex !== -1) {
            // Check there's another element following 'upgrade'
            if (upgradeIndex + 1 < n) {
                if (headers[upgradeIndex + 1].trim().toLowerCase() === "h2c") {
                    console.warn("HTTP2FilteredIncomingMessage: Stripping 'upgrade: h2c' header from incoming HTTP request");

                    // Lose the upgrade header
                    headers.splice(upgradeIndex, 2);
                    n -= 2;

                    // Clear the upgrade flag, which was set by parserOnHeadersComplete (lib/_http_common.js)
                    this.upgrade = false;
                }
            }
        }

        super._addHeaderLines(headers, n);
    }
}

const server = require("http").createServer({
    IncomingMessage: HTTP2FilteredIncomingMessage
}, app);

const io = require("socket.io")(server, {
    origins: "*:*",
    pingTimeout: 15000,
    pingInterval: 30000,
    transports: ['websocket', 'polling']
});

[darrachequesne]

For future readers:

Please check the documentation here: https://socket.io/docs/v4/server-initialization/#with-an-http2-server

Please reopen if needed.