Got an error reading communication packets

[metanovii]

version packages:
percona mysql 8.0.34-26
nodejs version v18.13.0
mysql2 3.9.3

config

  database: {
    type: 'mysql',
    host: process.env.THE_BEST_APP_INT_THE_WORLD_TYPE_ORM_MYSQL_HOST,
    port: parseInt(process.env.THE_BEST_APP_INT_THE_WORLD_TYPE_ORM_MYSQL_PORT),
    username: process.env.THE_BEST_APP_INT_THE_WORLD_TYPE_ORM_MYSQL_USERNAME,
    password: process.env.THE_BEST_APP_INT_THE_WORLD_TYPE_ORM_MYSQL_PASSWORD,
    database: process.env.THE_BEST_APP_INT_THE_WORLD_TYPE_ORM_MYSQL_DATABASE,
    autoLoadEntities: true,
    namingStrategy: new SnakeNamingStrategy(),
    logging: true,
    debug: true,
  }

Debug log

$ nest start prod
Add command: ClientHandshake
 raw: 0a382e302e33342d3236002e62c8000e441c256f2e272300ffffff0200ffdf1500000000000000000000670547297f53261858272a680063616368696e675f736861325f70617373776f726400
Trace: 
    at PoolConnection.handlePacket (/srv/app/node_modules/mysql2/lib/connection.js:433:17)
    at PacketParser.onPacket (/srv/app/node_modules/mysql2/lib/connection.js:97:12)
    at PacketParser.executeStart (/srv/app/node_modules/mysql2/lib/packet_parser.js:75:16)
    at Socket.<anonymous> (/srv/app/node_modules/mysql2/lib/connection.js:104:25)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Socket.Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
0 undefined ==> ClientHandshake#unknown name(0,,81)
Server hello packet: capability flags:3758096383=(long password, found rows, long flag, connect with db, no schema, compress, odbc, local files, ignore space, protocol 41, interactive, ssl, ignore sigpipe, transactions, reserved, secure connection, multi statements, multi results, ps multi results, plugin auth, connect attrs, plugin auth lenenc client data, can handle expired passwords, session track, deprecate eof, ssl verify server cert, remember options, multi factor authentication)
Sending handshake packet: flags:280687567=(long password, found rows, long flag, connect with db, odbc, local files, ignore space, protocol 41, ignore sigpipe, transactions, reserved, secure connection, multi results, plugin auth, connect attrs, plugin auth lenenc client data, session track, multi factor authentication)
0 13132334 <== ClientHandshake#unknown name(1,,161)
0 13132334 <== 9d000001cff3ba1000000000e000000000000000000000000000000000000000000000006c6561645f636f72655f6461746170686f6e655f7161001467b272d5f8dd1d87f05f62c1d42701f63e0fdc7e6c6561645f636f7265006d7973716c5f6e61746976655f70617373776f726400300c5f636c69656e745f6e616d650c4e6f64652d4d7953514c2d320f5f636c69656e745f76657273696f6e05332e392e33
 raw: fe6d7973716c5f6e61746976655f70617373776f7264000e441c256f2e2723670547297f53261858272a6800
Trace: 
    at PoolConnection.handlePacket (/srv/app/node_modules/mysql2/lib/connection.js:433:17)
    at PacketParser.onPacket (/srv/app/node_modules/mysql2/lib/connection.js:97:12)
    at PacketParser.executeStart (/srv/app/node_modules/mysql2/lib/packet_parser.js:75:16)
    at Socket.<anonymous> (/srv/app/node_modules/mysql2/lib/connection.js:104:25)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Socket.Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
0 13132334 ==> ClientHandshake#unknown name(2,,48)
0 13132334 <== ClientHandshake#unknown name(3,,24)
0 13132334 <== 1400000367b272d5f8dd1d87f05f62c1d42701f63e0fdc7e
 raw: 00000002400000000c010a096c6561645f636f7265
Trace: 
    at PoolConnection.handlePacket (/srv/app/node_modules/mysql2/lib/connection.js:433:17)
    at PacketParser.onPacket (/srv/app/node_modules/mysql2/lib/connection.js:97:12)
    at PacketParser.executeStart (/srv/app/node_modules/mysql2/lib/packet_parser.js:75:16)
    at Socket.<anonymous> (/srv/app/node_modules/mysql2/lib/connection.js:104:25)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Socket.Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
0 13132334 ==> ClientHandshake#unknown name(4,maybeOK,25)
query: SELECT VERSION() AS `version`
Add command: Query
        Sending query command: SELECT VERSION() AS `version`
0 13132334 <== Query#unknown name(0,,34)
0 13132334 <== 1e0000000353454c4543542056455253494f4e2829204153206076657273696f6e60
 raw: 011d000002
Trace: 
    at PoolConnection.handlePacket (/srv/app/node_modules/mysql2/lib/connection.js:433:17)
    at PacketParser.onPacket (/srv/app/node_modules/mysql2/lib/connection.js:97:12)
    at PacketParser.executeStart (/srv/app/node_modules/mysql2/lib/packet_parser.js:75:16)
    at Socket.<anonymous> (/srv/app/node_modules/mysql2/lib/connection.js:104:25)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Socket.Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
0 13132334 ==> Query#unknown name(1,,5)
        Resultset header received, expecting 1 column definition packets
 raw: 036465660000000776657273696f6e000ce00024000000fd01001f000005000003
Trace: 
    at PoolConnection.handlePacket (/srv/app/node_modules/mysql2/lib/connection.js:433:17)
    at PacketParser.onPacket (/srv/app/node_modules/mysql2/lib/connection.js:97:12)
    at PacketParser.executeStart (/srv/app/node_modules/mysql2/lib/packet_parser.js:75:16)
    at Socket.<anonymous> (/srv/app/node_modules/mysql2/lib/connection.js:104:25)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Socket.Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
0 13132334 ==> Query#unknown name(2,,33)
        Column definition:
          name: version
          type: 253
         flags: 1


Compiled text protocol row parser:

For nicer debug output consider install cardinal@^2.0.0
(function () {
  return class TextRow {
    constructor(fields) {
    }
    next(packet, fields, options) {
      this.packet = packet;
      const result = {};
      // "version": VAR_STRING
      result["version"] = packet.readLengthCodedString(fields[0].encoding);
      return result;
    }
  };
})()

 raw: fe000002000a000004
Trace: 
    at PoolConnection.handlePacket (/srv/app/node_modules/mysql2/lib/connection.js:433:17)
    at PacketParser.onPacket (/srv/app/node_modules/mysql2/lib/connection.js:97:12)
    at PacketParser.executeStart (/srv/app/node_modules/mysql2/lib/packet_parser.js:75:16)
    at Socket.<anonymous> (/srv/app/node_modules/mysql2/lib/connection.js:104:25)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Socket.Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
0 13132334 ==> Query#unknown name(3,EOF,9)
 raw: 09382e302e33342d323605000005
Trace: 
    at PoolConnection.handlePacket (/srv/app/node_modules/mysql2/lib/connection.js:433:17)
    at PacketParser.onPacket (/srv/app/node_modules/mysql2/lib/connection.js:97:12)
    at PacketParser.executeStart (/srv/app/node_modules/mysql2/lib/packet_parser.js:75:16)
    at Socket.<anonymous> (/srv/app/node_modules/mysql2/lib/connection.js:104:25)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Socket.Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
0 13132334 ==> Query#unknown name(4,,14)
 raw: fe00000208
Trace: 
    at PoolConnection.handlePacket (/srv/app/node_modules/mysql2/lib/connection.js:433:17)
    at PacketParser.onPacket (/srv/app/node_modules/mysql2/lib/connection.js:97:12)
    at PacketParser.executeStart (/srv/app/node_modules/mysql2/lib/packet_parser.js:75:16)
    at Socket.<anonymous> (/srv/app/node_modules/mysql2/lib/connection.js:104:25)
    at Socket.emit (node:events:513:28)
    at Socket.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Socket.Readable.push (node:internal/streams/readable:234:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
0 13132334 ==> Query#unknown name(5,EOF,9)
error Command failed with exit code 1.

messages from wireshark

M...
8.0.34-26./b..x.*K6NR....................X
.2|.S.ALLo.caching_sha2_password.....................................best_app_qa...f(......;d.P
.....Xbest_app.mysql_native_password.0._client_name.Node-MySQL-2._client_version.3.9.3,....mysql_native_password.x.*K6NR.X
.2|.S.ALLo......f(......;d.P
.....X........@.......best_app.....SELECT VERSION() AS `version`..........def....version....$..................
...	8.0.34-26.........3......#08S01Got an error reading communication packets

What am I doing wrong?

[BMO-tech]

Try reverting to 3.9.2. I was getting handshake errors today too (completely different setup than you, so no promises), and reverting from 3.9.3 back to 3.9.2 fixed it for me.

[metanovii]

Try reverting to 3.9.2. I was getting handshake errors today too (completely different setup than you, so no promises), and reverting from 3.9.3 back to 3.9.2 fixed it for me.

I rolled back the version to ^2.8.0 and it solved my problem. %)) And plugin auth mysql_native_password of course, sha2 does not work.

[pedrovanzella]

There's something weird with 3.9.3. I was getting SSL cert errors on RDS with the newest version too. I upgraded the certs in the cluster and version 3.9.2 works fine with the newest certs provided by AWS, but 3.9.3 doesn't.

[wellwelwel]

@pedrovanzella, can you show an example of your changes using the version 3.9.2?

[pedrovanzella]

@pedrovanzella, can you show an example of your changes using the version 3.9.2?

@wellwelwel there were no changes. I was running < 3.9.2 with the old RDS certs. I upgraded the cluster certs, and my code still worked. I upgraded to 3.9.3, and I started getting errors related to the certificates (they were bubbled up to sequelize, as a SequelizeConnectionError: unable to get local issuer certificate. I downgraded to 3.9.2 and I was able to connect again.

I mentioned this here in this issue because @BMO-tech had mentioned handshake errors, which is what this boiled down to - following the stack down I would see HANDSHAKE_SSL_ERROR.

[wellwelwel]

@pedrovanzella, @BMO-tech

I think the PR #2542 fixes this behavior, could you check it?

---

Closing this issue to continue this topic on #2541

[pedrovanzella]

@wellwelwel Just tested 3.9.4 and I still get the same behavior.

[wellwelwel]

Thanks, @pedrovanzella 🙋🏻‍♂️

We are handing over this responsibility to a dependency dedicated to it: mysqljs/aws-ssl-profiles#8

I'm thinking of "rebasing" the certificates again, just to make sure.

I trust this can be fixed soon.

[pedrovanzella]

@wellwelwel I commented on another issue already, but after some more testing, it does look like the issue is with RDS Proxy only. So it looks like you could just include the global CA certs in the file and we would be all set for all use cases.