DB Connection failure effectively kills queue daemon

[pdbreen]

• Laravel Version: 5.4.17
• PHP Version: 7.0
• Database Driver & Version: Aurora

Description:

Last night, each default DB queue daemon encountered a PDO connection refused error (these can occur on occasion within the AWS infrastructure). This left each daemon running, but in a useless state as any subsequent attempt to perform work resulted in an error attempting to invoke beginTransaction() on null. Restarting queues resolved issue.

Relevant stack traces follow:

Initial error (same error across every daemon instance)

Caused: PDOException SQLSTATE[HY000] [2002] Connection refused 
    vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:43 PDO::__construct
    vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:43 Doctrine\DBAL\Driver\PDOConnection::__construct
    vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:65 Illuminate\Database\Connectors\Connector::createPdoConnection
    vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:43 Illuminate\Database\Connectors\Connector::createConnection
    vendor/laravel/framework/src/Illuminate/Database/Connectors/MySqlConnector.php:24 Illuminate\Database\Connectors\MySqlConnector::connect
    vendor/laravel/framework/src/Illuminate/Database/Connectors/ConnectionFactory.php:183 Illuminate\Database\Connectors\ConnectionFactory::Illuminate\Database\Connectors\{closure}
    [internal] call_user_func
    vendor/laravel/framework/src/Illuminate/Database/Connection.php:882 Illuminate\Database\Connection::getPdo
    vendor/laravel/framework/src/Illuminate/Database/DatabaseManager.php:245 Illuminate\Database\DatabaseManager::refreshPdoConnections
    vendor/laravel/framework/src/Illuminate/Database/DatabaseManager.php:231 Illuminate\Database\DatabaseManager::reconnect
    vendor/laravel/framework/src/Illuminate/Database/DatabaseManager.php:165 Illuminate\Database\DatabaseManager::Illuminate\Database\{closure}
    [internal] call_user_func
    vendor/laravel/framework/src/Illuminate/Database/Connection.php:735 Illuminate\Database\Connection::reconnect
    vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php:140 Illuminate\Database\Connection::handleBeginTransactionException
    vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php:110 Illuminate\Database\Connection::createTransaction
    vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php:92 Illuminate\Database\Connection::beginTransaction
    vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php:192 Illuminate\Queue\DatabaseQueue::pop
    vendor/laravel/framework/src/Illuminate/Queue/Worker.php:236 Illuminate\Queue\Worker::getNextJob
    vendor/laravel/framework/src/Illuminate/Queue/Worker.php:101 Illuminate\Queue\Worker::daemon
    vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:100 Illuminate\Queue\Console\WorkCommand::runWorker
    vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:83 Illuminate\Queue\Console\WorkCommand::fire
    [internal] call_user_func_array
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:29 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:87 Illuminate\Container\BoundMethod::callBoundMethod
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:31 Illuminate\Container\BoundMethod::call
    vendor/laravel/framework/src/Illuminate/Container/Container.php:531 Illuminate\Container\Container::call
    vendor/laravel/framework/src/Illuminate/Console/Command.php:182 Illuminate\Console\Command::execute
    vendor/symfony/console/Command/Command.php:265 Symfony\Component\Console\Command\Command::run
    vendor/laravel/framework/src/Illuminate/Console/Command.php:167 Illuminate\Console\Command::run
    vendor/symfony/console/Application.php:826 Symfony\Component\Console\Application::doRunCommand
    vendor/symfony/console/Application.php:189 Symfony\Component\Console\Application::doRun
    vendor/symfony/console/Application.php:120 Symfony\Component\Console\Application::run
    vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:123 Illuminate\Foundation\Console\Kernel::handle
    artisan:36 [main]

Subsequent failures on every queue attempt to perform work

Symfony\Component\Debug\Exception\FatalThrowableError Call to a member function beginTransaction() on null 
    vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php:108 Illuminate\Database\Connection::createTransaction
    vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php:92 Illuminate\Database\Connection::beginTransaction
    vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php:192 Illuminate\Queue\DatabaseQueue::pop
    vendor/laravel/framework/src/Illuminate/Queue/Worker.php:236 Illuminate\Queue\Worker::getNextJob
    vendor/laravel/framework/src/Illuminate/Queue/Worker.php:101 Illuminate\Queue\Worker::daemon
    vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:100 Illuminate\Queue\Console\WorkCommand::runWorker
    vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:83 Illuminate\Queue\Console\WorkCommand::fire
    [internal] call_user_func_array
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:29 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:87 Illuminate\Container\BoundMethod::callBoundMethod
    vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:31 Illuminate\Container\BoundMethod::call
    vendor/laravel/framework/src/Illuminate/Container/Container.php:531 Illuminate\Container\Container::call
    vendor/laravel/framework/src/Illuminate/Console/Command.php:182 Illuminate\Console\Command::execute
    vendor/symfony/console/Command/Command.php:265 Symfony\Component\Console\Command\Command::run
    vendor/laravel/framework/src/Illuminate/Console/Command.php:167 Illuminate\Console\Command::run
    vendor/symfony/console/Application.php:826 Symfony\Component\Console\Application::doRunCommand
    vendor/symfony/console/Application.php:189 Symfony\Component\Console\Application::doRun
    vendor/symfony/console/Application.php:120 Symfony\Component\Console\Application::run
    vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:123 Illuminate\Foundation\Console\Kernel::handle
    artisan:36 [main]

Steps To Reproduce:

Unsure, but it seems simulating a connection failure during a queued job execution could recreate situation.

[howlittlegrass]

I met the same issue, my Laravel version is: v5.4.15.
It seems that: the DB connection was broken, which result in the daemon work of queue stopped working.

My stack trace as following:
[2017-04-24 09:51:06] local.ERROR: exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php:126 Stack trace: #0 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php(126): PDO->exec('SAVEPOINT trans...') #1 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php(113): Illuminate\Database\Connection->createSavepoint() #2 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php(92): Illuminate\Database\Connection->createTransaction() #3 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(192): Illuminate\Database\Connection->beginTransaction() #4 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(236): Illuminate\Queue\DatabaseQueue->pop('finance_settlem...') #5 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(102): Illuminate\Queue\Worker->getNextJob(Object(Illuminate\Queue\DatabaseQueue), 'finance_settlem...') #6 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\Queue\Worker->daemon('database', 'finance_settlem...', Object(Illuminate\Queue\WorkerOptions)) #7 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(85): Illuminate\Queue\Console\WorkCommand->runWorker('database', 'finance_settlem...') #8 [internal function]: Illuminate\Queue\Console\WorkCommand->fire() #9 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(30): call_user_func_array(Array, Array) #10 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}() #11 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure)) #12 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Container/Container.php(524): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL) #13 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Illuminate\Container\Container->call(Array) #14 /data/wwwroot/internal/surmob_finance/vendor/symfony/console/Command/Command.php(265): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle)) #15 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Console/Command.php(168): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle)) #16 /data/wwwroot/internal/surmob_finance/vendor/symfony/console/Application.php(826): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #17 /data/wwwroot/internal/surmob_finance/vendor/symfony/console/Application.php(189): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #18 /data/wwwroot/internal/surmob_finance/vendor/symfony/console/Application.php(120): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #19 /data/wwwroot/internal/surmob_finance/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(123): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #20 /data/wwwroot/internal/surmob_finance/artisan(36): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)

[pdbreen]

I don't have proof, but it looks like #19080 should fix this.