Hi, I've got a major site issue with my setup. It was all working fine until I got hit with a DDOS & moved to a DDOS reverse proxy which is a nginx front-end and a apache2 backend. There is a ping of ~170ms between the Frontend (Blacklotus Cloud) & Backend. I have been getting these a lot root@b020:~# cat /var/log/apache2/error.log [Fri Feb 17 18:02:27 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 18:02:51 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:03:05 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:04:19 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 18:04:30 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 18:04:30 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 18:04:55 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:04:56 2012] [info] removed PID file /var/run/apache2.pid (pid=27081) [Fri Feb 17 18:04:56 2012] [notice] caught SIGTERM, shutting down [Fri Feb 17 18:05:02 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations [Fri Feb 17 18:05:02 2012] [info] Server built: Feb 5 2012 21:35:42 [Fri Feb 17 18:05:02 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem) [Fri Feb 17 18:05:03 2012] [notice] Graceful restart requested, doing restart [Fri Feb 17 18:05:03 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations [Fri Feb 17 18:05:03 2012] [info] Server built: Feb 5 2012 21:35:42 [Fri Feb 17 18:05:03 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem) [Fri Feb 17 18:06:34 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:07:49 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 18:07:49 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 18:08:28 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 18:08:28 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 18:08:32 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 18:08:42 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:08:42 2012] [info] [client FrontendIP] (104)Connection reset by peer: core_output_filter: writing data to the network [Fri Feb 17 18:08:48 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 18:08:49 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:09:10 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:09:30 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:09:54 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:10:03 2012] [info] removed PID file /var/run/apache2.pid (pid=27574) [Fri Feb 17 18:10:03 2012] [notice] caught SIGTERM, shutting down [Fri Feb 17 18:10:04 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations [Fri Feb 17 18:10:04 2012] [info] Server built: Feb 5 2012 21:35:42 [Fri Feb 17 18:10:04 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem) [Fri Feb 17 18:10:40 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:10:59 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 18:11:11 2012] [info] removed PID file /var/run/apache2.pid (pid=27903) [Fri Feb 17 18:11:11 2012] [notice] caught SIGTERM, shutting down [Fri Feb 17 18:11:12 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations [Fri Feb 17 18:11:12 2012] [info] Server built: Feb 5 2012 21:35:42 [Fri Feb 17 18:11:12 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem) [Fri Feb 17 18:53:36 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 18:53:43 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 18:54:07 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:55:58 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:56:28 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:57:38 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:57:39 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:58:56 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 18:59:40 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 18:59:46 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:00:29 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:00:36 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:00:37 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:00:50 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:01:19 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:01:19 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:01:25 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:01:26 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:01:36 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:01:37 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:01:41 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:01:41 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:01:42 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:02:03 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:02:37 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:02:44 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:02:44 2012] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 14 idle, and 23 total children [Fri Feb 17 19:02:51 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:03:04 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:03:07 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:03:07 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:03:08 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:03:12 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:03:17 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:03:48 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:04:27 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:04:31 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:05:01 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:05:18 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:05:24 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:05:35 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:05:38 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:06:28 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:06:31 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:06:37 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:06:37 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:07:32 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:08:34 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:09:05 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:09:22 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:09:39 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:09:45 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:11:30 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:12:17 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:12:23 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:12:54 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:13:25 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:14:43 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:14:44 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:14:50 2012] [info] removed PID file /var/run/apache2.pid (pid=27971) [Fri Feb 17 19:14:50 2012] [notice] caught SIGTERM, shutting down [Fri Feb 17 19:14:51 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations [Fri Feb 17 19:14:51 2012] [info] Server built: Feb 5 2012 21:35:42 [Fri Feb 17 19:14:51 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem) [Fri Feb 17 19:14:55 2012] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 13 idle, and 15 total children [Fri Feb 17 19:15:21 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:15:22 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:15:24 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:15:48 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:15:55 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:15:59 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:15:59 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:16:06 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:16:08 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:16:29 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:16:30 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:16:32 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:16:36 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:16:41 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:16:54 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:16:54 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:16:54 2012] [error] [client FrontendIP] request failed: error reading the headers, referer: http://mysite.com/ [Fri Feb 17 19:16:55 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:16:55 2012] [error] [client FrontendIP] request failed: error reading the headers, referer: http://mysite.com/ [Fri Feb 17 19:17:01 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:17:01 2012] [error] [client FrontendIP] request failed: error reading the headers, referer: http://mysite.com/ [Fri Feb 17 19:17:07 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:17:13 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:17:41 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:17:44 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:17:48 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:17:48 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:17:55 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:18:01 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:18:15 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:18:25 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:18:42 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:19:00 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:19:14 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:19:25 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:19:41 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:20:00 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:20:25 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/ [Fri Feb 17 19:20:59 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:21:01 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:21:01 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:21:04 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:21:05 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/index.php [Fri Feb 17 19:21:07 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:21:09 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:21:30 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/index.php [Fri Feb 17 19:21:51 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:22:07 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:22:18 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:22:20 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:22:41 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:22:43 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:22:50 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:24:13 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:24:13 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:24:44 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:24:45 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:24:45 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:24:50 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:24:57 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:25:42 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:25:45 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:25:56 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:25:57 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:26:04 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:26:04 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:26:25 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:26:35 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network [Fri Feb 17 19:26:52 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:26:52 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:27:13 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:27:14 2012] [info] [client FrontendIP] Request body read timeout [Fri Feb 17 19:27:50 2012] [info] [client FrontendIP] Request header read timeout [Fri Feb 17 19:27:57 2012] [info] [client FrontendIP] Request header read timeout Code (markup): I have replaced actual frontend IP with >> FrontendIP The above error log reflects very vividly in website's load times and operations. Extremely slow! I've tried some different things, the website is a vBulletin forum. I disabled suhsoin using the .htaccess method and it showed a little bit of improvement but these errors are wreaking havoc. Nginx Config limit_zone ZZZ $binary_remote_addr 10m; upstream sarray { ip_hash; # server 1.2.3.4; //This 2nd server is load balancing but i have disabled it until i can fix the issue, otherwise changes have to made on both, I've tried using it alone but the error does not go away. server 5.6.7.8; } server { listen 10.10.20.30; server_name mysite.com; client_max_body_size 20M; listen 80 default_server; location / { limit_conn ZZZ 25; proxy_pass http://sarray; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } Code (markup): Apache2 config file http://paste2.org/p/1907478 (for better viewing) Also at times 502 Badass gateway error shows up (apache being unavailable to nginx) I have also tried to access the website using the back-end IP directly and it works blazing fast. I cannot seem to figure out why is the reverse proxy not working correctly, before this setup I used to have a reverse proxy setup as well how ever that was within the same DC. Can someone please assist me in debuggering off the errors Any help would highly appreciated!
I might suggest both http://httpd.apache.org/docs/2.2/mod/mod_reqtimeout.html and the usage of this in nginx.conf: ignore_invalid_headers on;