Apache crashing with PHP-FPM after period of time

Greetings!

I've setup a new server recently and moved a fairly busy forum to it (4.5 million posts, typically 500-1000 users online in any 30 minute period). I'm running Invision 4.4.4, which is reasonably resource intensive. Database (MySQL 5.7) running on one dedicated server, Apache running on another dedicated server. No other sites are running on the web server, and the database server only supports this one site.

While optimizing the server, I tried switching from PHP-CGI to PHP-FPM. Upon initially doing this, Apache would go down pretty quickly, apparently due to exhausting MaxRequestWorkers. I increased MaxClients to 2500 and MaxRequestWorkers to 2000. I also bumped up the PHP-FPM Pool Options a bit (Max Requests to 200, Process Idle Timeout to 10 and Max Children to 10).

This seemed to resolve the problem, but after about 12 hours Apache stopped responding to requests again, and I didn't see any active PHP-FPM processes. Restarting apache didn't seem to help, and it wasn't until I switched back to PHP-CGI that the server started responding to requests again. Moving from PHP-CGI to PHP-FPM is nice, as it dropped the load from above 3.0 to around 1.0, so I'd like to use PHP-FPM if I can. I tried this a second time (switching back to PHP-FPM), but after another 12 hours or so, it went down again adn I had to switch back to PHP-CGI.

I could try increasing MaxClients and MaxRequestWorkers further, but I don't know if that would help--seems like I might just be putting a temporary band-aid on a deeper issue.

I looked through the logs the last time Apache went down (or at least hung) and saw tons of messages like this:

[Wed Jul 03 22:20:16.028372 2019] [proxy_fcgi:error] [pid 12645] (104)Connection reset by peer: [client 110.249.201.217:10762] AH01075: Error dispat
ching request to :
[Wed Jul 03 22:20:16.028452 2019] [proxy_fcgi:error] [pid 12652] [client 111.225.148.7:11198] AH01067: Failed to read FastCGI header
[Wed Jul 03 22:20:16.028491 2019] [proxy_fcgi:error] [pid 12652] (104)Connection reset by peer: [client 111.225.148.7:11198] AH01075: Error dispatch
ing request to :
[Wed Jul 03 22:20:16.028512 2019] [proxy_fcgi:error] [pid 12651] [client 125.209.235.178:37891] AH01067: Failed to read FastCGI header
[Wed Jul 03 22:20:16.028586 2019] [proxy_fcgi:error] [pid 12651] (104)Connection reset by peer: [client 125.209.235.178:37891] AH01075: Error dispat
ching request to :
[Wed Jul 03 22:20:16.028620 2019] [proxy_fcgi:error] [pid 12150] [client 110.249.202.82:63448] AH01067: Failed to read FastCGI header
[Wed Jul 03 22:20:16.028642 2019] [proxy_fcgi:error] [pid 12527] [client 54.36.148.120:38628] AH01067: Failed to read FastCGI header

Code (markup):

Running PHP 7.2.19 and Apache 2.4.39. CENTOS 7.6 and WHM v80.0.20. Also running Redis on this server for caching. Machine has 32GB of RAM, and the site is running on a fast NVMe SSD. It's main task is just running the web server for this one site, plus email.

Thanks in advance for any insight and/or debugging advice.

 

Any slow database queries running when Apace goes down? If there's any mods being run, try disabling them one at a time when running PHP-FPM, leave it after each mod is disabled for 12hrs and see if it falls over, it could be a mod that doesn't like the current server config

 

Do you have enough resources on the server especially RAM? Normally, applications overload the memory which causes php workers hanging up. Increase the resources or an optimal configuration might help.