Desperate help configuring server with High Traffic website - 300 hits per minute

Hi guys i run a festival timetable website where people visit to check the correct times every day for a specific hour of the day. The website only reaches its peak for the 1 month every year.

For the last 4 years i've always moved my site from a shared server onto a dedicated server just before the festival starts. The dedicated server is good but it keeps on crashing due to the enormous hits reaching thousands within minutes.

Now my host is great but i think they're somtimes not up-to-speed with a few things - this is where i need your help.


Ok so my server was crippling and timing out with 500 Internal server errors, website showing just a blank white page. So my server was tweaked to the following settings:


Apache :

Timeout 300
ServerLimit 512
MaxClients 256
MaxRequestsPerChild 10000


Mysql :

max_connections=1200
table_cache=1800
key_buffer_size=256M
wait_timeout = 25
interactive_timeout = 25
connect_timeout = 25
query_cache_size=128M


I tried accessing the site from my phone or another computer it would not load, it's as if the website was cached onto my local computer hence why it was loading fine for me. From other new connections, friends computers and phone it would load so slow with the website looking broken and images and things all over the place not fully loaded.


So i explained to support staff that only i and a few others can view the website, but not new visitors. I even tried accessing the site via a proxy browser and it would not load, so the support staff tweaked the server down slightly which fixed the problem a l. This was her response:


On further checking, the MaxClients value set in Apache configuration exceeded the limit.

---
root@host [~]# grep MaxClients /usr/local/apache/logs/error_log
[Thu Jul 19 10:08:13 2012] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Thu Jul 19 14:31:15 2012] [error] server reached MaxClients setting, consider raising the MaxClients setting
---

We have increased MaxClients from 256 to 300 in Apache configuration file. Then I have enabled KeepAlive directive to reduce the number of Apache connections in the server. Also, I have reduced MaxRequestsPerChild from 10000 to 1000.

MaxRequestsPerChild 10000 to 1000
Timeout 300 to 30
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 1000

Then I fine tuned following Mysql variables in the server.

1. thread_cache_size from 32 to 16
2. table_cache from 1800 to 1024

The load average in the server is fine. I am monitoring the server and shall keep you posted."



I then asked for Nginx to be installed installed which didnt make a huge difference, I was asked by somebody to ask my host whether I had php-fpm and nginx as proxy this was my hosts response.


"You don't have php-fpm, your PHP is set to run as Apache module which is the fastest method.

Yes, nginx is setup as reverse proxy on your system - it accepts all incoming requests, it handles static files on its own by reading them directly from the filesystem and it passes requests to dynamic content back to Apache web server which generates the content and return it to nginx for further delivery."


Because i was still having problems for the website to load, the "max number of idle processes" were increased so my new config became:

Apache config changes:

StartServers 10
MinSpareServers 10
MaxSpareServers 30
ServerLimit 256
MaxClients 256
KeepAlive On
KeepAliveTimeout 3


nginx config changes:

comment out the "disable_symlinks if_not_owner;" line.




The website loads fine off peak, on average i receive upto 70k thats 70 thousand hits per day, with 1 hour being the most busiest. The website counter display's a few thousand current visitors viewing the website, this is displayed for the last 2 hours.

Even with a dedicted quad core server and nginx installed my website doesnt seem to be able to cope with the basic of traffic? My host seem to be exhausted not so sure what to do and i'm now left in limbo desperately trying to tweak my server for the peak hour which i receive once every day.

I have Munin server monitor installed, i am happy to show you screen grabs if anybody can help. I really dont know a great deal about managing servers and am desperately in need of help.

I would really appreciate if somebody could shed some light and put me out of my misery.


Many thanks and i hope to hear from somebody soon.

 

300 hits per minute? That's quite a site you have there.

 

This only lasts for an hour per day as people come to check a timetable for the festival before the traffic subdues.

 

Something like this is nearly impossible to troubleshoot just from descriptions posted to a forum. Too many variables in such a general request.

That said, 300 requests per minute is not really a lot. Did you mean 300 per second?

You should explore the 500 errors to try to get an exact cause. Error logging should be able to point to the problem. Sounds like memory but could be a lot of things. All of the maxclients and mysql configuration changes will make no difference if you don't have enough memory in the box.

One option is to switch to LiteSpeed vice apache. It is faster and more efficient than apache and, with built-in caching (2-CPU license & above), it can handle many thousands of concurrent connections.

Lastly, instead of a dedicated server you might want to explore a Cloud-based solution that scales automatically. Auto-scaling cloud solutions are generally more cost effective than VPSs and dedicated servers because you're not wasting money paying for RAM you don't need 90% of the time.

 

I'd also consider putting all static files (images,css,js etc) onto a CDN like rackspace or cdn77.com, leaving your server to just serve the php/html