Some operations cause Valet+ to hang completely

[SanderAtom]

It's hard to pinpoint what is going on, but after certain actions involving Ajax calls (I think), Valet+ becomes completely unresponsive. There is minor php-fpm usage when I open a page, but very short then nothing and eventually it times out with a 504 Gateway Timeout.
Very rarely it started working again after a few minutes without doing anything. Mostly I have to restart php and it works again. Sometimes it hangs again on the same Ajax call immediately after, but mostly the Ajax call is handled quickly and without problems after restarting php. Until I open some other pages, save something and try the Ajax call again, it breaks. This happens with a Drupal 8 project.

~/.valet/Log/nginx-error.log:

2017/08/22 10:15:50 [error] 5708#0: *5648 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://unix:/Users/sander/.valet/valet.sock", host: "project.dev", referrer: "http://project.dev/"

Nothing in ~/.valet/Log/php.log

I see this in /var/log/system.log, but I think it's just a warning because php is always running:

Aug 22 10:07:14 nm2d-02345 com.apple.xpc.launchd[1] (homebrew.mxcl.php70): This service is defined to be constantly running and is inherently inefficient.

Does anyone have any idea which php/nginx settings I can try to solve this?

[SanderAtom]

This happens now every time Drupal has to create new images based on image styles. When I change the image style and load the page, php hangs when trying to create the new images. I have to go to the image link directly to get the image to be created. After they are created, the page loads find and quick. Not sure if this is because these scripts are executed multiple times at the same time.

[Tristan-N]

@SanderAtom This also occurs on Magento 2 installations, so it's not a Drupal only issue.

[SanderAtom]

Ye but less often than certain drupal actions. Also only restarting php resolves this issue, restarting nginx has no effect.

[SanderAtom]

@Tristan-N do you have anymore information about what kind of actions cause this complete crash for you in Magento2?
When this happens it won't even get to server.php so might have something to do with the valet.sock? Not sure though how that works.

[SanderAtom]

I see this in /usr/local/var/log/php-fpm.log:

[04-Oct-2017 13:40:17] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
....
[04-Oct-2017 13:50:58] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 5 total children
[04-Oct-2017 13:50:59] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 6 total children
[04-Oct-2017 13:51:00] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 7 total children
[04-Oct-2017 13:51:01] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 8 total children

etc, etc.. Whatever I keep raising, the other settings will max out again. So if I raise pm.max_children it'll warn about being busy. When I then up the min/max and start_server numbers, eventually I'll run into pm.max_children problems again.

It seems specifically bad when there are a ton of requests, like unmerged css/js + image files etc when those calls might have to generate files as well.

I kept raising the numbers, now it reads:

pm.max_children = 50
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 40

I get rid of the warnings in php-fpm.log but it'll still break at some point without any error and give 504's until php is restarted.

[Tristan-N]

@SanderAtom I don't have real examples. The problem is I am not sure what is happening. When I, for example, want to create a backup from the backend (instead of CLI) it takes up to like 20 seconds before getting the 504 time-out. However, when I just keep waiting suddenly a backup-file is created in the folder it should be. So it seems the processes are still running but 504 is shown in browser.

Funny enough, I don't have anything in the errlogs either.

[SanderAtom]

Ok, but when you get this 504, can you still open other pages? Or you have to restart php before you can do anything on the website?

[Tristan-N]

I do have to restart php to be able to open pages again (I use valet restart for no particular reason, other then lazyness)

[SanderAtom]

Using higher numbers for children and start/min/max servers does lower the rate at which php crashes, but I keep getting warnings to increase them. Are there related settings we can try, something like buffer size or something?

[SanderAtom]

I see this in php.log, not sure if it's related..

[24-Oct-2017 09:16:16 Europe/Amsterdam] PHP Warning:  apcu_add(): Unable to allocate memory for pool. in /Users/sander/.composer/vendor/weprovide/valet-plus/server.php on line 61
[24-Oct-2017 09:16:16 Europe/Amsterdam] PHP Stack trace:
[24-Oct-2017 09:16:16 Europe/Amsterdam] PHP   1. {main}() /Users/sander/.composer/vendor/weprovide/valet-plus/server.php:0
[24-Oct-2017 09:16:16 Europe/Amsterdam] PHP   2. apcu_add() /Users/sander/.composer/vendor/weprovide/valet-plus/server.php:61

[Neodork]

@SanderAtom and all. Noticed same behaviour (504 gateway timeouts, on pages doing a lot of requests) when installing weprovide/valet-plus. laravel/valet v2.0 or higher does not cause this behaviour (if you install it without PHP installed). Spend a lot of time getting to the root of the problem here, but eventually a college of mine cracked the problem. He has no Github account so I will post on his behalf, credits to him. But the likes are mine!

Cause: Opcache conflicts with APC(U) which causes PHP-FPM children to stay open after a request. Which eventually causes your machine to run out of resources, one way or another.

In /usr/local/etc/php/{php-version}/conf.d/z-performance.ini

Change:

apc.enabled=1

To:

apc.enabled=0

End Result:

...

[apcu]
apc.enabled=0
apc.shm_size=512M
apc.ttl=7200
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.enable_cli=1

...

Then restart your PHP service. But to be sure, just restart everything: valet restart

NOTE: Fully uninstalling the APC PHP extension breaks Valet so no need to try that. Instead disable APC following the instructions above.

[SanderAtom]

@Neodork thanks for that information, I'll look into it. A quick scan reveals this thread about PHP-FPM crashing or processes lingering due to APCu turned on, so that seems promising. Although I do remember turning off features before, including APCu, without any direct improvements but I can't say for sure. I might setup a test situation where this crashing occures, without a full webshop system and all the caching layers around it, and see if simply turning off APCu solves the problem.

[SanderAtom]

I'm fairly certain this is the bug we are experiencing. Since turning off APCu 2 weeks ago, I have not had any php-fpm/socket etc hanging like it would before.

[Neodork]

@SanderAtom Thanks for getting back to this. Since you're confirming this solves the problem I will continue to PR this today. Thanks!

[SanderAtom]

Well just turning it off isn't really a fix for the problem. Do you have more information about this conflict between OPcache and APCu? Is it something other people are working on and we can add the fix for it later?
On the other hand, what's the benefit of having APCu and is it important enough to wait for the fix or resolve the conflict? Otherwise we just need to completely remove APCu.. or OPcache .. ?
Turning it off for now is ok to at least prevent new people from experiencing this conflict, but I feel we need a better solution long term.