Recover from idle_in_transaction_session_timeout

[tmtron]

Expected behavior

After an idle_in_transaction_session_timeout, the used connection will be working properly when used again from the pool.

Actual behavior

After a statement timeout in transaction, the used connection will be "broken" and new queries will not work. The error-message is Client has encountered a connection error and is not queryable

Steps to reproduce

I've added a test-case to my project fork

Environment

• Version of pg-promise: 10.2.1
• OS type (Linux/Windows/Mac): Windows 10
• Version of Node.js: 12.12.0

Related

• Stackoverflow: How to handle idle_in_transaction_session_timeout?
• similar issue: Statement timeout in transaction returns a broken connection to the pool #566

Notes

I thought that adding another check to isConnectivityError for code 25P03 (see postgres error codes) should work, but it does not. In this case the err parameter has no code property, but a message string.
A nasty workaround for now is to check this message string:

    if (!code && err.message && typeof err.message === 'string') {
        if (err.message.includes('is not queryable')) return true;
    }

[vitaly-t]

This is from invalid transaction usage, as the author indicated so himself.

[tmtron]

I don't see why this issue is closed: IMHO it is clear that pg-postgres is currently not handling the idle_in_transaction_session_timeout error correctly: i.e. it must be treated as a connectivity error and the "broken" connection must not be reused. I hope we agree on that.

So maybe you can give me some thints on how to get the error-code (and not just the error-text) in the isConnectivityError, I'll try to come up with a PR.

[vitaly-t]

I thought the issue was sorted.

Just so, this library does not handle idle_in_transaction_session_timeout. It is a lower level for node-postgres perhaps.

[tmtron]

Well, the specific issue in our app is fixed.

But I think it's just too easy that such an issue can slip into the application code again in the future. And then it would be great to have the idle_in_transaction_session_timeout as a safety-net so that the app can recover automatically. Then the app would still be usable until we find the issue in the code and install the new version.

I'll check node-postgres and their issue-list when I find some time.

[vitaly-t]

This is an unusual issue, and is the first time I ever hear about it.

And if you can detect it for a specific transaction code, you can force the physical connection to be released, using the following synthetic approach:

async function specialTransaction() {
    const obj = await db.connect(); // manually allocating the connection
    let idleTxIssueFound = false;
    let txData;
    try {
            txData = await obj.tx(async t => {
                 // execute the transaction, and detect the idle issue, if possible
                 idleTxIssueFound = true; // if found an issue
            });
    } catch(e) {
      } finally() {
          // passing true into method `done` will force physical release of the connection
           obj.done(idleTxIssueFound);
           return txData;
        }
} 

This is more of a proof of concept though.

[tmtron]

After checking some code and issues of node-postgres, I think my initial assumption was wrong. We will never receive error-code 25P03 from the database. Instead we only get the error-string from node-postgres::

query.handleError(new Error('Client has encountered a connection error and is not queryable'), this.connection)

I think whenever this error happens (for whatever reason - not only when idle_in_transaction_session_timeout is used), pg-promise must remove this connection from the pool because it is broken.

Since node-postgres only returns an error string, and no error-code or error-class that can be checken, the only way to handle this case is to check the text of the error-message.

There are many related issues in node-postgres:

• #1942 node-postgres handles server disconnection differently on macOS and Linux
• #1790 "Client has encountered a connection error and is not queryable" without further info
• #1611 Connection terminated unexpectedly

and in node-pg-pool:

• #673 Connection terminated due to connection timeout / Connection terminated

[vitaly-t]

It's a tough one to tackle. I mean, there is not good fix from this end, by the sound of it.

[tmtron]

I can also make the test work without checking the error-string and use client._queryable instead

e.g. change line 59 of function poolConnect from
client.release(kill || client.$connectionError);
to :
client.release(kill || client.$connectionError || !client._queryable);

That is for sure better, but maybe the underscore in the variable name _queryable indicates that this variable is private and should not be used?

[vitaly-t]

This does look like a hack-fix, i.e. should work in the current implementation of the client, but may get broken later, as we use a private variable.