Hello everyone, It's my third time changing the host and I am curios to know how much traffic your Apache server support and which is the hardware you are using. I noticed that when several robots come to my site at once, the site hangs up. But the most consuming process seems to be MySQL, not Apache. Please let me know if you are dealing with the same problems.
I have a site with 3000 -> 7000 uniques a day, sometimes hits around 130/sec when spidering is active, but no problems with that, so we use a Pentium IV 2,4Ghz with 1024MB RAM for that and it's load is normal around 0.18 so no problem... If you need advice or managed service for this let us know, maybe we can arrange something!
130 hits/sec ? This will force Apache to fork many child processed to handle this amount of requests. Each child eats up to 25-30 Mb of RAM. Are you serving dynamic or static content? What database server are you using ?
I have a dedicated in a data centre at a hosting co, it's just an old celeron with 512MB ram, 80 gig hard disk, 10meg ethernet. It's serving about 2gig / day, with about 6000-7000 unique's a day. (appox 16 sites, 3 busy ones, all php/mysql driven) you can make even an old machine serve up fast if your good at tweeking apache/php/mysql. Another server I administer (PIV, 2gig ram, 100meg ethernet is serving over 1mil banner ads a day and over 1T traffic a month) serious stuff. might look at load balancing soon
Almost all apache servers are set to handle 500 or less simultaneous connections at the same time. So far all the stuff written in thread is not relevant to actual simultaneous connections in any given sec. You can find this in your http.conf file if you set it to higher amounts and you actually get more than 500 in a sec then your server is destined to spiral down out of control. remember you can have thousands of people visiting your site a day and hour a minute but still not breach actual 500 connections in a given sec. they would literally have to hit the page exactly same time. If you have download site this can be different game. hope this helps
Concurrency slow-downs can be a quite difficult bug to track down. You do some load testing on your website by firing off request after request after request and have no problems so you make it live and suddenly the whole thing falls over. One user can get any page back in 1 second but 10 users have to wait 10 seconds each ! The trick is in finding which part of the process is taking the most time and why it is taking so much time. A common problem is having a single database table that is both read from and written to on every page view while using MySQL's MyISAM engine. The MyISAM engine cannot do reads and writes at the same time and whichever one comes second will have to wait for the first query to finish before it can start. If more queries of the first type appear while the first query is still running, they will queue up behind the write and will have to wiat until both the first and the second queries have finished before they can start. We once had a problem with two load-balanced servers trying to write to the same file which was shared over NFS. NFS doesn't handle file locking very well and when this happened it would cause the Apache process in question to lock up for a few minutes. The most likely course of action that will solve this is to wrap this PHP code around certain commands: $start_time = microtime(TRUE); mysql_query("SELECT * FROM `blog` ORDER BY `date` DESC"); /* This is your existing code */ $end_time = microtime(TRUE); echo("<!-- MySQL blog query took: ".round($end_time - $start_time, 3)." seconds. -->\n"); PHP: Browse your site when it is going slow and see which one of those code snippets caught the locking. If the problem is MySQL, look in your MySQL slow logs. Use: mysqldumpslow -l /path/to/mysqlslow.log Code (markup): to find out which queries spent the most time waiting for other queries to finish before they could start. I can't tell you how to fix the problem until we know exactly what the problem is but I do know of some solutions. Switching a MySQL database engine to InnoDB will usually solve any concurrency problems. If you don't have any concurrency problems like the sort I have just mentioned then it might just be necessary to get better hardware. One technique that can help significantly is to switch all of your static content (images, js, css, etc.) out on to a separate server with a different domain name. As for what we use for our websites: Four Sun T1000 servers for static content and SSL decrypting, five Sun X4100 servers as PHP webservers with memcached running locally, four Sun X4100 servers running MySQL - one as a master with two slaves and one as a standalone master - and one Sun X4500 server sharing the document roots and home directories out over NFS. The T1000s have 8GB of RAM and the rest have 12GB of RAM. T1000s have Sparc processors with 24 virtual cores running at 1000MHz each and the X4100s have two dual core AMD processors running at 2.8GHz per core. The X4500 has the same AMD processors as the X4100s but it's main feature is it's disks... 48 of them! The websites in question serve varying amounts of traffic but the top-end ones do 3 million unique IPs per month. The bandwidth sits around 95Mbps between 9am and 11pm and drops off sharply after that. Even with all that hardware, however, all 70-odd sites can be taken offline by a badly written MySQL query.
I have dedicated server apache2 suse linux 10.2 currently some 24'000 uv/d and currently have without any problem mixed traffic 2 MySQL db served sections of site plus flat files some 1+ million hits per day average and ten thousands very large ( 200-800 kB ) files 17-27 GB ( average - max ) per day NO problem at all !!! but with a fraction of above traffic, I was kicked out of shared hosting at 1and1 last year fall even tough I used less than 2% of my booked/paid package! BTW I had in the past 10+ years many host changes and almost all had to do that I used too much bandwidth/resources the pain for my EX-hosts is my joy when I look at my adsense data hence there is a time for shared hosting and one day definitely is a time to move to your OWN root server !!! latest when a host says you suck too much cpu/bandwidth = means to you are TOO successful to suffer from shared hosting pain!